Subscribe
Share
Share
Embed
Developers building a software business on our own terms.
Ben: And we're back.
Josh: We're live.
I have something, I have something to read here and I, it's hard to believe that it's been this long, but, Andrew Mason on Twitter. Says, I can't say how badly I want this, which is, FounderQuest, to return,without sounding weird, but it's been 699 days and I'm still subscribed.
Hoping it returns, listening, listen to it all multiple times and have highlights backed up in my notes. If I ambush you with a mic at a conf. Can you just start again, please? so this is, obviously we're dBen:oing this, specifically for Andrew, and maybe, just to avoid an ambush. But, yeah, we'll see.
Ben: Yeah.
Josh: Maybe he'll still, maybe he'll still ambush us, sometime. That might be nice.
Ben: Ambushes are always welcome. Yeah,
Ben: 699
Josh: that it's been 699 days and this is episode 100
Ben: Episode 100.
Josh: FounderQuest 100 in the 11th year of Honeybadger. This is our, we've been doing this 11 years. can you believe that?
Ben: I can't believe it.
Josh: So we're a long distance runners.
Ben: yeah, I think we, we look a little older than last time we recorded.
Josh: I know. Yeah. what does that like a couple of years? I, it feels like it doesn't feel that long to me, but like looking back on what, everything we've been working on the past couple of years, like it's, yeah, it's been a blur, but, yeah, time flies.
Ben: Yeah, it really doesn't feel like that long, I guess we just had our heads down building stuff and, living the dream that, we just didn't notice time going by.
Josh: You could say the same thing just for the business in general. It does not feel like we've been doing this 11 years, but,
Ben: That, that's, that
Josh: just think of that we were a decade younger when we, when we got together and with star and started this thing, it's just, it's wild.
Ben: you didn't have kids when we started Honey
Josh: I know such a different, place.
yeah, now I've got a five and a seven year old.
Ben: Yeah. Yes, they
Josh: kids in college.
Ben: Yeah, my, yeah, my kids are now in college. It's crazy. I guess we're off the top though. We got it. We got to talk about why it's just two of us and why Star isn't with us.
Josh: Yeah.
we're going to do a kind of a, a recap state of the union.
Ben: Yeah, we'll catch people up.
Josh: quest, 100 episode state of the union, Honeybadger. And we're going to, what we're doing moving forward.
Ben: Yeah. We'll catch up on all the things that have happened in the past few years. Excruciating detail. We'll be here for six hours. So buckle up.
Josh: we'll try to, we'll try to keep it, try to keep these quick. I know people like their, their 30 minute episodes,
Ben: Let's try to keep it quick. Yeah. and as always, if anybody, if you're listening and you have questions about anything we talk about, you want us to go more in depth on something, feel free to reach out to us on Twitter or, wherever, however, us and can get in touch with
Josh: Mastodon, whatever.
Ben: happy to talk more about whatever that you find interesting.
Josh: Uh, the major news is, uh, Star is no longer with Honeybadger.
Ben: So in March, 2022, we bought out Star's portion of the business. Star decided that she wanted to move on to do some other things. And, we decided that, Josh and I decided that we were ready to take on the business, and that we could go ahead and move forward.
Josh: We wanted to, we had things we still wanted to do.
Ben: Yeah, we, even after 10 years, we still have big dreams. I think of things we want to accomplish. And of course our GitHub backlog is probably hundreds deep at this point.
Josh: Yeah. like the real question is who, who was the smart, who made the smart choice?
Ben: Totally. Yeah.
Josh: that the jury is still out on that, but we'll see. maybe we can get back together with star for like founder quest, like 400 and when we're all in rocking chairs and do the retrospective.
Ben: yeah, because the terms of the deal, like we didn't have, the cash on hand, Josh and I didn't have all that cash to, to be able to, cause we, we evaluated the company, we value the company based on, typical multiples that were going on in the SaaS marketplace at the time, we've, we talked to people like FE International and Uh, others out there who, you know, what's going on with acquisitions and that sort of thing.
So we're not experts on that sort of stuff. so when we were starting to have conversations about the buyout, we thought, we got to find out how much the business is worth and basically use the guidance from those kinds of business brokers to find out what the typical range is.
And I think we settled on 4X, if I remember correctly, four times revenue.
Josh: Something like three, between three and four.
Ben: yeah, I can't remember. we had, we. haggled of course.
it's always a negotiation, no matter how much you like someone, it's, business makes everything a negotiation. Yeah. And when, when you're in the position that Josh and I were, you're funding it yourself. we didn't have, we didn't have big investors backing us. We didn't have a whole bunch. we had to negotiate as much as we could down. And so I want to negotiate as much as she could up.
And I think we had, it came to a happy compromise in the end.
Josh: Yeah, I think it
Ben: so yeah, so now we, so the way that we're working at now is that we are, working to, to pay off the debt that we have to star and, uh, some, someday we will be free of that. But, but it's been, it was a kind of a rough transition.
I think there was a few months there where it was just like, wow, we had to do a bit of a reset, how to see what the business was going to run like without the third co founder.
Josh: Yeah, a lot was, everything was different.
Ben: Yeah. Yeah. There's a lot up in the air. but I think we got through it
Josh: I think we were definitely nervous about like servicing that debt, that's a big check we cut every month. And, at the time it was scary. Like, you know, we're, we're signing up, signing on, signing back on to this business. and, on top of it, we have to consider that we have this extra, whatever chunk of the.
The balance sheet that, that isn't available. but in hindsight, I think our fears were a little bit, unfounded, at least luckily, like I think we've, you know, it, it doesn't feel like too much has changed in terms of just our [00:06:00] day to day being able to work on the business and, make money, pay our employees, have some, some, a nice profit margin, like it's.
I think it's worked out as well as it could, for everyone, someone in our, the situation that the three of us were in. So
Ben: Yeah, I agree. I think, one of the things you said made me think that, yeah, it really was a gut check, I think all three of us were sitting there thinking, do we want to continue doing this? Do we want to keep, we've been at it for 10 years. Do we want to invest another, whatever years our lives in this?
And, uh, yeah, but, uh, yeah. I agree. I'm with you, like enjoying what we're doing. it's, I haven't, since then I haven't thought, oh, that was a really bad decision. no, yeah, I've been really happy with, continuing to work with you
Josh: We just, we have a business and, business is capable of paying, paying large loans. And yeah, we just just, just another thing you think about.
Ben: Yeah. Yeah. So that was definitely a big transition. I think another big, personnel change, so we'll fast forward to 2023, about six months ago, we brought on another employee, so Roel has joined us from the great white North of Canada and, started, yeah, about six months ago.
And it's awesome to have him join us. he also has a couple of kids or three kids. And so we were continuing the tradition of everyone at Honeybadger happens to have some little kiddos running around. and, but, so we're else developer does a bunch of rail stuff has been diving in, it's done a fantastic job, love having him here.
So that makes a grand total of five again. three, we got what, Josh, you and I are, I guess we're still developers, right? We still write some code from time to time.
Josh: some code
Ben: we got Ben who's doing marketing for us, Ben Findley, and we got Kevin and Roel who are also doing development and it's, it's been fun, it's been a lot of fun having us all work together.
Josh: and then our extended contracting. Team is a little bit larger. It has fluctuated a little bit, but, we have some long term, long term folks that have worked with us that we like, like working with and Pangratios and, Bethany until recently, uh, Shalva. Um, so it's nice to have, you know, we have a little community of people,
Yeah. And on the iOS side, I was just talking about this yesterday, like we still have Andre working with us. He does our mobile apps and can't forget Andre.
Ben: with us like, what, eight years or something, part time as we need
Josh: And everyone else we're forgetting, by the way, I'm
Ben: not mentioning everyone, but,
That's true. they're a lot.
Josh: yeah, so that's, yeah. so yeah, Roel actually joined us, in, Atlanta, earlier this year for, RailsConf. and it was cool to get the team, some of the team together and, get some face to face time, that was our first, that was our first time meeting Roel in person, so we had some good.
Good dinners and, uh, time to hang out but I don't want to jump too far ahead because actually, RailsConf, 2020 was, a pretty epic RailsConf for us. and probably for FounderQuest as well, because I forget where we left. We talked about this forever back in the pandemic, but at one point, I didn't look up the episode numbers. So I can't be cool and drop the episode number, but you can go like search. but we were planning, the RailsConf that got canceled. I think it was right. was it RailsConf 2021, that got canceled or was it 2020?
Ben: I can't remember, but yeah, it was, yeah, it was,
Josh: forever go again.
Ben: I guess it's probably 2020, the first one that got canceled because of COVID yeah.
Josh: yeah.
we had these big, this like grand scheme to, have a, what I think we were like going to do, have a lounge, um, at the conference. I think we were calling it the indie lounge, for, you know, kind of make a, group thing for like indie creators. and then, we were, planning to rent, ground control, local cart arcade, In Portland here that we really like to have a, a Honeybadger bash.
And, we were very sad that got, that all fell through when the pandemic made, you know, all the conferences were canceled. 2022 was the year that RailsConf finally made it to Portland. after all that, Heartbreak and we were able to, do all of that. So that was cool to see that come, get some closure on that.
And it was a lot of fun. Like it was really cool. having like a central place, in the conference exhibit hall with the lounge, it became like a hub for people. People still mentioned it to me. And, and then the party was awesome. Arcade games and, great turnout.
Ben: Yeah. Ground control. I love it. I love it. It's fun to have an excuse to, have a deductible party at ground control.
Josh: right. Yeah. So I feel like we did pretty well. We did pretty well at rails. Cause we did, my home, my hometown justice, for, yeah, for representing.
Ben: Yeah. when it's on your home turf, you got to go all out,
Josh: Yeah, you got, that's. Yeah, that's right. so yeah,RailsConf, this year it was in Atlanta and, we decided to be the lounge sponsor again. We didn't do a big party, but, the lounge has been fun to get to meet a lot of people there.
And it's, it fits our style. Like we're not very like salesy people, in person. So it's nice just to have a place where you can kind of talk, meet people and then the things. Work comes up more naturally and you can still talk about that stuff, but there's not you're not like pressuring people into demos or that sort of thing, which is just not my jam.
Ben: Yeah. We are definitely not, not salespeople who are at the booth, right? Trying to get your business card, right?
Josh: Maybe we should be, I don't know, but, I think it's all
Ben: we could try something different if we want to.
Josh: It's fine.
Ben: and then, RailsWorld is going to be in, in Toronto next year, right?
Josh: Toronto. Yeah.
Ben: And that is Rowell's hometown. So I guess we'll have to pull out all the stops again for that one.
Josh: Yeah. I got, I will, we'll have to make it to, to Toronto. Yeah.
Ben: I haven't been to Toronto. I've been to, let's see, Vancouver. Oh, yeah. I went to Vancouver with the first International Rails conference. Pretty sure that was the official name of the conference. That was,
Josh: was in Vancouver.
Ben: was that 2015 or 2016? I don't know. It was before the first, rails Comp. That was by done by O'Reilly, which was in Chicago.
Is that Chicago?
yeah. So that was in v Canberra and Vancouver. It was a lot of fun and, it was, I remember it was over, Easter weekend. Or, or right up to, because I was there and, went out for like breakfast on Good Friday and everything was shut down on Good Friday. I'm like, what's going on?
I knew it was Good Friday, but I wasn't used to things shutting down for that, but that's the way it is in Vancouver.
Josh: Really? I didn't realize that. They shut down for a good Friday.
Ben: So that was fun. Okay.
Josh: did you get breakfast or didn't you? I didn't,
Ben: I don't remember,
all I don't remember now.
Josh: Yeah, yeah, people are dying to know. I know right now everyone's on the edge of their seat.
Ben: So yeah, so that was fun. RailsConf 2022. yeah. Love Ground Control. we should do a team building activity there. We should,
Josh:
Ben: block out the night again, like we did there and just have just the five of us hanging out. That would be cool.
Josh: Yeah, that would be fun.
Ben: I guess we can talk about some of the building stuff that we did. Some of the business things, things that we
Josh: we also, we did ship some things in
Ben: Yeah, we weren't
Josh: with all that going on. It wasn't just all, yeah, it wasn't all heartbreak and parties.
Ben: we had a big update to status pages. You want to talk about that?
Josh: yeah, so I think that was, that was the big, I think that's like the big thing we did, feature wise that year, which was, status pages, which in the past we, we had a feature cause we, we have, we do uptime monitoring at Honeybadger and, Uh, you know, common thing that people want to do with uptime monitoring is like relay that status, are things up or down to their customers.
And so in the past, we've had some requests, can we, can we have a status page where it shows like if my sites are up or down, and we had this kind of limited feature where you could, enable a, It was I don't know, a very simple status page, for a specific uptime check that it would show.
I don't remember if it was a, was it a group or was it just individuals? I think it was just one. it would show the status and like the historical performance of that check. and that was okay, but, it was nothing like, like status page. what is it? Status page? I oh, That was acquired by Atlassian, or some of the other like more full featured status pages where you can get pretty complicated with them, like showing all of your, like your infrastructure, and that sort of thing.
so this is a feature that we've had, people, multiple people have requested over the years is like a, Actual like status page feature. so that's what we built. We built a very simple version, to start out, which was like, you could basically like customize, some of the, theme, add a logo and add some, change some, some of the colors on the page.
and of course, show multiple checks, which was, I think that's like. What version one of this was, or, yeah, version one of the new status page, feature. and then since then we've added some, we've added some additional things. You can actually, you can also show your check ins. So if you've got like jobs that should be running, you can communicate if they have been running on time, say you have a billing task, that is supposed to run like, or.
Have you sent out the email that, email notifications for the day, that sort of thing. so we've been adding additional, types of, statuses that you can display. and then, yeah. And on some other, I know we've been this year, we've been adding even more. So you probably got a better list of that in your head.
[00:15:00] Yeah. When, when Roel joined us, Getting him into the app and we decided to take a look at status pages again. And to see, again, I, I mentioned earlier, we have that, the hundreds of items in our backlog and some of those, a lot of those items are feature requests that come in from time to time.
Ben: And we had a number of feature requests for status pages, like various features to roll up into that, bigger feature. And, so we got real well started on that and we've pushed out some of those things, like some more customization options. Shalva did some work on that as well, like being able to tweak the styles of the page and that sort of thing.
but also we have features like, requests, like being able to have subscribers to your status page, to be able to get updates, via email and that sort of thing, to have password protection to, there are a few other things, and those are coming out pretty soon. I think we're just about wrapping up kind of a bundle of features for that.
you'll see in the next few months, more and more features coming out for that. That will get us closer, I think, to what Atlassian the status page offers you so that, you don't [00:16:00] have to go to a third party to have your status page. You can just have that hosted with Honeybadger, simplify things a bit.
So we may not, we, I don't know that we'll get to the point where we're like feature parity. That's not really our goal. our goal is more to give what our customers want and not, they're not necessarily asking for all those features that Atlassian has, but, yeah, so we'll see more, more to come on that.
Josh: I'm really looking forward to being able to like do email subscriptions and, and some of those other channels outside of, just notifying Twitter or whatever.
One of the, one of the other features that we, That we added pretty quickly after, in 2022, after we, we launched the initial version of status pages was, incidents. so you can, in addition to like just automated, if your checks are up or down, you can add, Specific, updates about either outages like incidents or, if you have like scheduled plan to maintenance, for example, you could, schedule an incident that it would then like pre announce, that you're going to be like.
Say updating your database or moving your database or something. And then, at the time that you have that [00:17:00] scheduled, it would start to go into like, you know, this is the, this is the planned outage. And then if you have additional, updates, if it doesn't go according to plan, as those things sometimes do not,
Ben: It never happens.
Josh: of course, you could, of course, add additional updates to keep everyone in the loop.
since,yeah, maybe your official channels are not, not, up at the moment.
Ben: I love that feature. I love being able to post updates. That's probably the most. So we have our status page hosted with, Atlassian status page because of course, If all of our stuff is down, we don't want to have our own status page hosted on our stuff, which might be down. So we use the third, we use, somebody else for that. And, the biggest feature that I use from, Atlassian status page is that, that incident thing. something's going wrong. I love being able to communicate to customers that, Hey, there's a problem. It solves the problem of having a bunch of people emailing us saying, Hey, I think something's broken.
And it's yeah. You can see the status page. It is broken. or the maintenance thing is also really cool. Like I love being able to say, Oh, we're going to do a database [00:18:00] maintenance this weekend. you can expect there's going to be, 15 minutes of downtime or whatever, so it gives people a heads up that, yeah, I need to plan my usage of the app around this outage, which is, which we know about ahead of time.
So I love that feature. So glad we have that in our status pages product.
Josh: Users and customers love it too. I think people really appreciate the transport, that transparency. and yeah, even when that's unplanned outages, like it's, it, it's never good to have downtime, but, how you respond to it, It can really make it, you know, it can kind of spin a bad situation into a, a positive in some cases where if you handle it the right way, so if you don't handle it the right way, then it's just, it's just two, two terrible situations, but
Ben: Yep. And let's see other big developments. we started working last year on a project that's pretty big, but, not ready to chat about that just yet. maybe later on in the episode,
Josh: later in the podcast, I, yeah, you have to keep listening for the next couple of minutes and we might get there. We'll see, but just, we have been working on it for, for most of this year,
Ben: yeah, I think it's been
about a year.
Yeah. But we did do a lot of work on, not necessarily the product per se, but on the business, the marketing side, the website, we've done a lot of work with that. And, we worked with some vendors, conversionary experts was one of them and we're continuing some work there.
Really helping us, nail down our messaging and, make improvements to the site so that people actually know what we're about when they land there. we're not marketing experts. And so bringing in some consultants was really helpful for there. And one of the things I, I loved about the process was that, they would, interview our customers.
And so we had a number of rounds of, having people sit down with them and say, okay, let's take a look at this bed page and how does it hit you? And what do you understand about it? And what do you think? And that was just so illuminating, to be able to get some third party perspective on,it doesn't make sense at all, or I don't know what you're talking about here, or, that sort of stuff.
super, super helpful.
Josh: Yeah. You reminded me, that's one of the things I would like to, I'd like to resume at some point here is, and I'd [00:20:00] like to, like, in work, be able to do that more internally. I think we should do more customer interviews and research. and Yeah, we, we did a lot, like at the start around the start of 2023.
I think it's one of those things that you don't necessarily need to be doing it all the time. Of course, like you, you don't want to be like, asking people too often to sit down with you. especially since we found that, And this is great in our case, we have like this, this group of people, in our customer, out of our customers that are extremely willing to give us feedback and sit down with us, but at the same time you want to like, value their time and everything.
So I think periodically though, it's good to do a round of research and kind of sit down and see where people are, where their head is. and that's, I think working with conversion rate experts, was really, Really great in that respect to just because you mentioned we aren't, we aren't marketers.
and so being able to see their process, which is very they try to take a very like scientific approach to things where, everything is backed by data and,[00:21:00] as you know, to the, as best as it can be. and we're not just making. Blind decisions, or just based on what we're feeling at the moment, but we have actual,kind of research and, and data to, to back that up.
that's not always been our strong point. we're much more gut people and, I think we continue to be, but, bringing a little science into the. Into the, process is never a bad thing. so we're trying to, we're trying to get a little bit better at that.
And, and also just like they're admittedly like crazy expensive, like consultants are not cheap people. if small bootstrap company, like we're not, we're not working with, this sort of, consulting, the thing like. All the time. but it's when we are, it's a great opportunity to learn like how they do what they do.
And then hopefully we can internalize some of their, some of their processes in, the company, like in our. Marketing that we do. So that's been like my, what I've been working [00:22:00] on, for some of this year is, trying to replicate some of that ongoing, like just experimentation and doing some research and trying, having an idea and trying to see if you can validate little ideas that might move the needle.
In small, small directions.
Ben: Yeah, I think one of the, one of the things that we love about Honeybadger is that the churn that has always been super low, like customers who join with us and start, using us, they stick around forever. but we've always felt like we could do better on the acquisition of customers and the activation of customers.
getting, people in the door, checking out the site and getting people actually signed up and actually using it, getting over the hurdle of actually installing the gem and sending errors and really fully activating. And so you did a big chunk of work around onboarding and helping improve that process.
And that's been, I think that's been really useful. Yep.
Josh: there's never ending work there. it's, yeah, there's a lot of, a lot of, low hanging fruit.[00:23:00] but yeah, we've tried a bunch of things this year and in the spirit of experimenting with. Different marketing ideas. we recently, just took away having a free plan.
Like we, we were like, we've had a free plan, for I forget like years, a really long time. I can't remember that when we, had not had one. So I was like, it might be a good, Why don't we just see what it does to the numbers. If we don't have, if we don't offer a free plan, like maybe, I've heard people being of people being surprised, that, like they've been doing this one thing all along that has been like, whatever cannibalizing sales or, you, it's easy to make, make a decision like five years ago, especially when you're like this small and, there's so many things to think about.
So you make a decision and then just don't think about it for five years. So it's good to revisit those things periodically and just like start from the beginning and see, see if things are different. So we tried, we tried that. and we actually, we had, I think we had some movement there in terms of Converting trials.
Like [00:24:00] it's a lot easier to convert trials, for example, when you don't have the longer sales cycle of a freemium. Type funnel. we confirmed, I think we, we knew this in the past, but it's just, it's good to remember, these are two funnels that we can't treat the same way.
So if we're going to have a free plan, which we do, we just actually recently just brought it back. but we're bringing it back in a different way than we did before. We want to like actually treat it like a real, Like a real, a channel for, for converting people, but it's very different from trial activation.
we're trying to be more systematic about, about that sort of thing. So I think those two experiments have been pretty successful so far.
Ben: Yeah, it was, eye opening to me. maybe I just should have realized this, but, I don't know, maybe I'm just dumb, but seeing that the freemium. path is really different than the trial path, right? It really is a different activation thing. And, we, we've tried different, incentives to get those people.
And those two, two different, channels basically moving, through the funnel. so it was, yeah, it was really eyeopening to me to see how [00:25:00] different those two use cases are funnels,
Josh: Yeah, if we're talking, mistakes that we've made, I think that one of them, if we've ever made a mistake, I don't, the jury's still out, but if we have, one candidate would be, just not treating the, just ignoring the free. Users too much, and not really having a good visibility into like how they like the, how they think and the psychology, like the psychology is different, of someone who's not willing to pay when they start, but maybe they are down the road and all the inflection points are different.
And I still don't fully understand that as well as I'd like to, but that's part of this process is like, we want to understand if we're going to have a free plan, we want to understand those people as well as we do our normal, our normal, like more business, like trial, customers who are like looking for a solution, like now and have a business they want to monitor or whatever.
So yeah, lots to talk about there. Maybe we can, we can revisit this as we learn [00:26:00] more in the future. You
Ben: Yeah. And, on my end, I've been doing a lot of work on, product keeping, keeping Kevin, Kevin's been working a lot on this super secret project, which we'll talk about in a few minutes. and, I mentioned talking about adding features, getting familiar with the app and from. In addition to that, I've been looking at the op situation.
Josh: So we're always, making improvements on the backend of Honeybadger. I, I'm an old sysadmin, I'm a tinkerer. I can't let things just be right. I, even though it's working, I want it to be working better, always got to be optimizing your bash scripts
Ben: that's right. So we, we completed this year, completed a migration from, Amazon's EC2 to Amazon's ECS.
So super late to the Docker game. Containers. Yeah, for real. Like, uh, you know, I'm, I'm old school. Like I believe in servers and, you know, blinking lights and stuff. And so when we, when we moved to Amazon back in 2007, long time ago, we just moved to VMs. that was fine. And I didn't really want to get into the whole container thing.
I just didn't love [00:27:00] Docker. I wanted to not give up my VMs. But eventually over time, I've, I think I've been one over and I'm more of a fan now, a Docker than I was back then. And so now we're at the point where we're running everything on ECS and love it. Should have done it years earlier, but it is great.
Like the, a big part of it is, our sidekick processing, right? That's where most of our magic happens. And, spinning up a container obviously is faster than spinning up a new. EC2 instance, right? And it's just great. Like we, we have seen, our latency go down. We've seen the backlog spikes go down.
we've handled, traffic spikes of 5X, 10X, 20X, maybe even more, I don't know, I can't remember, without
Josh: Yeah. We hada really big one recently and you were just like, yeah, it's just I forget what it was, but it was an eye popping, multiple.
Ben: yeah, I think it was like 20 or 50 X or something. And that happened within a course of minutes, right? two minutes later, we're doing a heck of a lot more [00:28:00] traffic.
And then, 10 minutes later, it shuts off again. And, ECS is great for that. It just scales basically infinitely as far as we're concerned. And we have the, we know we're watching all of our backlogs and it's just automatic. And every now and then we see an alert pop up on our Slack channel and we're like, Oh, CPU is high and go and take a look.
And oh yeah, it's scaling. It's doing what it's supposed to do, so that's been fantastic. Highly recommend. A plus would do again.
Josh: it was cool. recently,I had a project that involved, adding a new, like named sidekick, queue. And which is not something we do frequently, you usually want to keep your, keep the queue names or the queues you're working, to a handful, I'd say, but, we had a new one to add and, we, uh, Like getting it, getting like something working, it was much, much easier than it would have been with EC2, because you were able to just what create like a new task that would just work that specific queue and it didn't even affect like the rest of our infrastructure.
it's that's that would have been a major change in the past, I feel like, or it would have felt much more major, but this made it feel more of like, well, if it, you [00:29:00] know, if it's isolated and, We can we can optimize it from there, but like, it's not going to affect the rest of our pipeline, which is, I think that's the big one.
It's just like, you don't want to introduce like a new queue that like blocks the other important things that are happening.
Ben: Yeah.
Josh: that was fun.
Ben: That was fun. And, uh, you know, one of the experiments we were able to do also, was adding Kafka into the mix. So we love Sidekiq and, we're, I don't think you'll ever be able to pry it out of our cold dead hands, but, we also use Kafka for sending a bunch of, messages around. we've. Changed some of our pipeline to accommodate that.
And we've just been able to also experiment with some more event driven programming, which has been fun. being able to add that to the mix without having to worry about interrupting what's already working has been pretty fun.
Josh: Yeah. Yeah. that's cool.So we also are no longer using Hamil. which is a complete departure because we're talking like heavy ops stuff and this we're talking, this is like whatever, a template language, but, this was a big deal for us because we have been, I don't know what the word [00:30:00] is, but, like we have hated Hamel for for years.
Ben: it was a love hate relationship,
Josh: it was a love hate relationship, but it's over the years as we've as we've scaled with it, as we've added more views to our rails app and had to edit old ones. and, and also, I think there's been like. Some perform it's like things have been faster since we switched to ERB.
but we had been kind of like gradually switching over to ERB for a number of years. And that, that also was just a annoying because every time you'd have to go edit one of the old views, you'd have to switch back into like camel brain, which is now a thing now. Term I'm going to use. and, and yeah, this is like a small, like one of those minor, like maintenance things or tech debt things, but, we've done a few of those this year as well.
so that was a lot, we, we had over, I think we had over 300 views remaining that we, we can converted. I did most of the reviews. I didn't do that. We had one of our, one of our contractors worked on the [00:31:00] actual conversion, with a mix of automated and. just, file by file, change migration, but, reviewing it is also, a little bit of a mind numbing, like my eyes were crossing every day, like after reviewing like camel side, side by side diffs basically.
So I like lived and breathed Hamel diffs for a few weeks. and then, yeah, with some other little like tech debt, Repayments like, we might, we upgraded our bootstrap, which we still use. and I don't know if we'll ever get to use tailwind or any, anything cool on the front end, but, it's a, it's modernized bootstrap and it's, it's working pretty well, so
Ben: Yeah, it's been fun to see getting out those updates out there and making things just a little nicer for the DevExperience internally.
In fact, I saw in the microcomp community, yesterday, this morning, something I saw someone talking about, there was a question about,what do you do in December and the end of the year when it gets really slow and people are on vacation and, how do you [00:32:00] handle, the response was keeping lights on and without over, overloading the people that are still around and things like that.
And, one of the answers to that question was, as a person running a Shopify app. And so their Q4 is typically pretty quiet because in general, e commerce people don't want to make changes in Q4, you got Black Friday, you got Christmas. so I just want things to be the same. don't break anything.
and so since their business is so quiet at that time, he said, what they do is they. A lot of maintenance tasks. So they'll go in and just focus on polish, focus on quality and, pick some things that have been bugging them or that they, some tech debts that they want to get rid of. And they'll just go, to the nth degree on getting that thing done, but at a high quality, solution.
So I thought that was pretty cool. we don't, we do something like that in December. We have a, what we call a hack week. So the first couple of weeks of December, we just, we, instead of focusing on a tech debt, but we do. Spend some time on that. But instead of restricting it to that, we basically encourage everyone to work on something fun.
Josh: to just like a hack week, you pick a project and you go and work on it. And maybe it's, learning a new program language. Maybe it's doing advent of code and a language that you're not familiar with. and sometimes it works out that we have work on work projects as well. you might do I was going to say, I have a feeling we'll be working on work projects. this, I don't think we'll be able to convince people not to work on work projects this year, just because we're excited about what we're working on.
Ben: Yeah. I suppose we should talk about that. we've been building up this whole time.
Josh: Yeah, in any case, like I'm, I, our approach to holidays sounds much better, like than tech, just like doing bug bashes. just doing maintenance work is, at the holidays. It's not the most exciting thing. So yeah. Hack week. I always appreciate.
Ben: Well, you know, uh, on what we have reminds me, we have experimented this year with, our on call rotation and using that. So what we do at Honeybadger is, every developer is on rotation is on call for one week, and then we rotate that every week. And so now there's four of us on call. So that kind of works out pretty well.
Once a month, we can expect to be doing that. And part of [00:34:00] our on call rotation is doing customer support. we don't do much phone support, but we do, a fair amount of email support. And so one person, that person who is on call is tasked with, handling those emails as they come in, doing the triage and, handling it if you can, or, passing it off to someone else if you can't.
And, one of the experiments that we've done this year with that is to also treat that time as, A time where you can set aside some of your project work because you might be distracted by customer support requests coming in and just focus on the low hanging fruit, the little, bugs or whatever fixes that you feel like are been annoying you for the past little while.
And I think that's worked out pretty well. I've enjoyed that freedom to just set aside my normal work and look at, a maintenance task or, a little ops thing that's bugging me or whatever.
Josh: Yeah. Yeah. It's not a bad approach to that maintenance stuff. since you're already distracted and juggling, juggling things,
Ben: Right?
Josh: we'll have to, there, I think there's a lot of things where we'll be able to pull out of here and focus a little bit more on them in future episodes.
that's, that's always exciting that we have new things to talk [00:35:00] about on the podcast, if we're going to keep doing this, which, I think we want to, and, maybe that's a good, We can talk about, what's next for Honeybadger. What's next for founder quest. we probably should not keep people waiting too long on, on this, this new effort we're working on.
right.
you want to talk about that,
Ben: Yeah, I'll take it. I'll take it away. so we're working on a new, a big new enhancement to Honeybadger. We're calling it insights and it's, a structured logging slash event logging. addition to Honeybadger that allows you to track whatever kind of events you want. Now, most developers are going to look at this and say, Oh, structured logging.
I will send my application logs and that's cool. And yes, that's cool. you can do that, but it's not just about logging. it's about sending in your events as well. So the first thing we've done is we've put in all the Honeybadger events is going into this new database. We're using Clickhouse, which is freaking awesome.
You should definitely take a look at it. If you haven't ever played with it. But we're using ClickHouse to make it really easy to query a whole bunch of data. So let's say all of your, all your Honeybadger errors, all [00:36:00] of your uptime checks, all your check ins. So you can query across that data and your application logs and whatever events you want to send in.
I don't know, maybe you're tracking the number of user signups or. How long it takes someone to activate from a free, you know, plan to, uh, to a paid plan, like we do, right? and having all of that data in one place where you can query it and you can build dashboards around it, you can build metrics from your logs.
Let's say you want to track the page serve time and that's in your logs. You can. Run some aggregations that show you how that changes over time to all kinds of cool stuff. And we're calling that insights and that is coming pretty soon now, we've been working on it, like we said, for about a year, and,
Josh: The, the
backend in our defense, the backend is kind of, kind of insane. Like I'm in the scale that we're, anticipating is, orders of magnitude different from, Yeah, so
Ben: Yeah, we, yeah, and to be honest, but a little sneaky, in fact, it didn't really take us a [00:37:00] whole year to build and add this into Honeybadger. We actually started off on a different path and we had to make a pivot, which is why it took us as long as it did, but we're going to be able to use that original work that we did.
And we'll talk about that more later. I'm not going to talk about that today, but,
Josh: we got to give people a reason to tune in for the next episodes.
Ben: but the short story is while we were on that path doing that thing, we decided, Oh, you know what, we need to bring this on the Honeybadger. We need to have this feature because it would be so freaking cool to be able to query our error data or, to be able to see these kinds of charts on the fly.
we gave you some charts inside of Honeybadgers today, but. we probably don't give you everything that you'd like to see, or exactly the way you'd like to see it. So insights is going to be that, and, we're pretty, pretty excited about it.
Josh: yeah. In hindsight, it's something we had to bring into Honeybadger. Like just, it's one of the, just because it's, you want, if you want like the, like a holistic monitoring platform, um, this is kind of, I think it has potential to become like the central hub of everything in Honeybadger.
And, and it [00:38:00] will, like you said, it's going to let us do some really cool things, like even going back to status pages, for example, we can imagine being able to set, alerts on, like metrics going over thresholds or that sort of thing. And you can add that to your status page.
And, whereas, yeah, if you're in the honey, you're, if you're using Honeybadger for Most of your monitoring, it's just going to make everything so much simpler than say, you're using like, cloud watch and am in AWS and, like a third party status page per like status pages IO.
it's just gonna, it's gonna make, it's gonna make, give everything the Honeybadger treatment, basically make it dead simple to use and, integrated. So pretty excited about that.
Ben: Yeah. And as we've gotten into it, at first we thought, Oh, we'll add this feature, right? And then we realized, you know what? Oh, this is actually changing the whole product. Like we can really centralize everything around this kind of thing. so yeah, you'll be seeing some big changes pretty soon, both to the product and to the marketing side, as we Try to figure out exactly what this means, for our customers.
And I'm sure we'll learn along the way, some new things that we're [00:39:00] not even considering today, but yeah, we, we, we love using this like internally. We've been using it for a while and it just, every time I use it, I just get a smile on my face. it's awesome. again, like Kevin was the main one behind a lot of this work and he just knocked it out of the park.
It's been
Josh: we'll have him, we'll be doing a full episode on this or multiple full episodes. I think, there's a lot of cool stuff we can talk about here. maybe we'll even talk Kevin into joining us for some of that. we'll see. I've been working on them, um, but, I wanted to say one, One of the, I think unique, there's a lot about this that is unique from what, like the competitive landscape.
but one of the things that we landed on kind of as we've been working on, like, how are we going to position and market this feature, is like you, you kind of hinted at the fact that you, this isn't just an engineering tool. Necessarily like you, you're obviously like, it's going to be pretty like heavy engineering use with our customers.
Like you're going to send your logs, your request, you're going to have like performance dashboards for your,[00:40:00] request data and that sort of thing. but I'm excited about sending like a, like marketing. Data to it or like events from the application, like customer data, for example, because, then we can basically like use it as a tool to query, if we're, if we need to look something up for like a customer support request, we can go and query like what we're, you know, what was this person doing or like, what's their usage for a specific feature, um, that sort of thing.
So I think there's like all kinds of use cases that aren't. Limited to just engineering, which is, I think that's, I think it's an interesting approach, in a, like monitoring tool like ours. I haven't really seen anyone else speak to that use case. and I think that, I think there's.
Potentially some opportunity there for us to kind of stand out and differentiate ourselves a little bit. It's like, you know, Honeybadger does not just have to be only, used by. Developers, for example, like you could give, maybe this is an opportunity to give, someone [00:41:00] else, like a technical user on the on the marketing team access to, to Honeybadger.
so we'll see where that goes. but it's just an interesting. thing that we've landed on, and it'll be fun to explore.
Ben: Yeah. And, even if David recently had his keynote, we were talking about the single developer framework or making that one developer more productive. And, even in the case when you're solo, having all this stuff in one place where everything that you care about in your web app is in one place where you can query and chart and yeah, I think that's going to be
Josh: Maybe that's why I'm, maybe that's why I really like that jumps out at me as a founder, like you are all of those roles
and, um, and as a, I think develop like technical or like developer founders will really appreciate that because, I like, I don't know about you, but I've tried to use pretty much every marketing tool on the planet and none of them have ever been able to quite satisfy what I'm looking for.
I always appreciate more like developer oriented tools and tooling that I can use to cobble together my own. [00:42:00] My own solutions, which whether it's to my detriment or not,I like, we like to build our own marketing tools. but this is gives you another option for, another way to query events and have dashboards and stuff like that for customer data.
Ben: Yeah. And you can be tracking page views. Like you said, you can be tracking a product usage, like
Josh: Oh yeah. Analytics. We could pipe our analytics in here.
Ben: yeah. Clickstream analytics, you name it. Like any event that can happen in your app, you can shove into insights and query it, see it right along with all the rest of your data that you care about for your app.
Josh: Exciting.
Ben: and then, for the FounderQuest, what's the future for FounderQuest? We talked about the future for HoneyBadger, FounderQuest, we are definitely going to keep going. Oh, this is not a grand finale departure episode. No, this is a reboot episode. we don't really know exactly yet what our schedule is going to be.
It might be that we just can't keep up the weekly thing. so maybe it's going to be, I don't know, bi weekly, maybe it's going to be monthly. Yeah. we'll figure it out. We'll keep you posted and we'll see what makes sense. But, but yes, we're going to be around. We're not gonna, [00:43:00] we're not gonna leave you hanging for another 700 days waiting for the next episode.
Josh: Yeah. I think, we have, we have a lot of things that we could try with the podcast and, Yeah, it's good. It's good to get back. Good to get back in the podcast chair. I'm enjoying this.
Ben: I think one thing that could be cool is an idea I just had and we've had, interview style for, episodes in the past, right? We've brought on some people, you know, be kind of cool is to bring on people who want to interview us, right? Who want to ask us about something particular at Honeybadger or whatever, that would be fun.
So if you're interested in that kind of thing, feel free to reach out and let us know.
Josh: Yeah, of course. any, we're open to any and all ideas. questions, things to talk about. yeah, et cetera. Hit us up.
Ben: I think overall our plan is just to keep the same chit chat format where we're gonna talk about the stuff we've been working on, the stuff that we're coming out soon, and experiences, things we've learned in that course, basically more of the same.
Josh: yeah. why change if it's working?
So where can people, I guess we got to do the whole, we always do this, at the end of every episode, we're [00:44:00] like, Oh, I don't know, what are, how do we end this thing? we're gonna, we're gonna keep the, we're keeping the format. Um, so where can people find us? Uh, so we're a founder quest on Twitter.
there's I'm not going to call it the other name, but what it's Twitter.
Ben: It's still Twitter to
Josh: if you're on Twitter, you can find us at FounderQuest. I think you can also find Ben and I on Twitter, through there. It's, at Stimpy and at HeyJoshwood on Twitter. And, and then for all of the other socials, since there's
like 50, 000 of them. Now, you can find on founder quest podcast. com. I'll make sure that I list all of our other outlet, like links, links there. So yeah, go follow us on social media. and please review us on Apple podcasts or Spotify or wherever you're listening. I think that will help us get, new listeners.
Hopefully some of our old listeners are still with us. so yeah. Thank you for tuning in and thank you for, for making it, 699 days without a founder quest episode. at least Andrew, we know one person's listening. thank you, [00:45:00] Andrew. And, thank you all of our other, listeners who have stuck with us over the years.
it's good to be back.
Ben: thank you so much for pinging us on a regular basis, asking us to resume our episodes. We,
Josh: Got to
Ben: do appreciate it. Yeah. we, we love hanging out and love hearing that it's fun for you too. So thanks.
[00:46:00]