Developers building a software business on our own terms.
Josh: That's how we like to do it.
Ben: We just like to jump right in there.
Josh: We have a guest with us today who is not John Nunemaker, for a change. We have Adam McCray from Judoscale is here. And I'm really excited about this. Adam and I got to hang out at RailsConf recently in Detroit, and that was a lot of fun. Glad you could be here, Adam, to get all official, to get all serious here, suddenly.
Adam: It's fun to be here. I appreciate it. And yeah, finally, a break from John. I bet everybody's ready for a break from John.
Ben: How did I miss you at RailsConf, Adam? I don't remember seeing you.
Adam: I caught a glimpse of you at one point, and then never saw you again. So I don't know—I don't know how we missed each other.
Josh: This is why you never leave on the last day; you stay for the whole time, because I think it was the last night that we went out to dinner, wasn't it?
Adam: Yeah
I think it was after Ben left. Somehow, I missed you like the whole first half of the conference too, so I was happy to, , or I might've like seen you like from afar or something like that, but yeah, glad we got to go out to dinner.
Ben: Yeah., I was regretting that choice, but it was really the lesser of two evils, right? My return flight was either at—going to be at 7 p. m., or at 7 a. m. And since, you know, that was Eastern time and I was heading back to Pacific, I was like, “Well, I'd much rather fly out at 7 p. m. than 7 a. m.”, but yeah, I didn’t.
Josh: I took the 7am and let me tell you, it was—it was rough.
Adam: Yeah, neither of those options sound fun.
Ben: No, no, so I guess I should have just gone full tilt and gone 7pm the next night, right? Just spent an extra full day in Detroit.
Josh: Normally I do the thing you did and I leave like the afternoon, but there's been a few times I've regretted it, so I figured I'd try staying for the whole, you know, through the whole, like last night, this time.
Adam: And Josh, you just traveled across the country again for Blue Ridge, right?
Josh: Yeah. I was just a Blue Ridge Ruby last week. Thursday and Friday were the days of the conference. It was really—it was a great conference. I had never been to Asheville, North Carolina. Did I hear that you guys were there, one year, or last year?
Adam: We did go last year, yeah, we did. Three of us, or four of us, did a road trip from Columbus, Ohio to Asheville, North Carolina, which was like—it was like a six or seven hour drive, but when we weighed out the flight options, it's like, "Well, it's going to be about that much travel time anyway, so let's just drive.”
Josh: Yeah. Plus you got to like a road trip with a bunch of software engineers. That sounds like fun.
Adam: It was fun, yeah. And it was an awesome conference. So yeah, I'm glad you got to go this year. I'm glad they did it again this year, because it was a great time last year.
Josh: Yeah, they did a great job. Joe and Mark organized it this year—and Jeremy got a break, but he still gave us a great keynote at the end of the—on the last day and it was really fun. I'll have to put our—I did a write up on our blog of a recap of the conference. We'll have to put that on the show notes.
Adam: Yeah, I read through it this morning. it was a fun recap. Yeah. Ben, do you have any—do you have any other conferences you're going to be heading to this year?
Ben: Actually, speaking of travel and conferences, I have a bit of a quandary. So I am planning on being at Rails World, and so that's in September. But I was also thinking, “Maybe I want to go to Business of Software this year”. That's every year on the East Coast. I think it's in Raleigh now. It used to be in Boston every year.
I was like, “Well, maybe I'll do that”. And so I got an email from them and I was like, “Let me go check that out”. And so, it's the same week as Rails World—but it's the beginning of the week. So I technically could fly out Sunday night from Seattle to Raleigh and then wrap up the conference on Wednesday, like lunchtime, hop on a plane to Toronto to be there for Thursday morning Rails World and then fly home on Saturday from Toronto back to Seattle.
So I'm trying to decide do I really want to do that? Because I typically I, you know, conference travel is two or three days and you're done and that's kind of my sweet spot, so a whole week of traveling is not really my usual so I don't know. But yes, I'll be at Rails World for sure
Adam: I'm exhausted at the end of a conference. I cannot imagine bouncing from one conference to another.
Josh: Yeah, I've never done back to back.
Ben: Are you planing on going?
Adam: I will be at Rails World, yeah, and I'm super excited about that because I just hired my first full time developer at Judoscale. Carlos Antonio de Silva, who's also a Rails core contributor. He and I used to work together at You Need A Budget or YNAB.
Josh: I was going to say YNAB, yeah. I'm a big fan of YNAB.
Adam: But yeah, we met each other in person once when we were at YNAB, but, , that was pretty short. So, I'm excited to see Carlos in person. He's gonna be at RailsWorld as well.
Ben: Nice.
Josh: That's awesome.
Ben: Yeah, back to back might be a bit of a bad idea. Like, I imagine by Friday I would be pretty fried. So, maybe—maybe I don't do that.
Josh: I remember I did one time, I think I did a back to back week. So I was like, I went one with the first week and then I was home for a few days. And then I went—I had another one the next week. I think that was probably that, like, 2013 or 14 stint that we did where we went to a lot of events and that was pretty tough, I recall, but you should definitely do it.
Adam: You should, you should at least definitely do RailsWorld., and I won't, and I won't miss you this time.
Josh: Yeah, I actually I have to—we’re going to have to end this by 11 today because I'm going to try to snag a ticket there. They're dropping the last batch of tickets for Rails World. And I was kind of on the fence when they did the first one. So I'm going to see if I can grab one, and if I can, then I'll be there. And otherwise, uh, I don't know, maybe I'll come hang out in Toronto anyway.
Adam: I mean, the first batch was gone in, like, 20 minutes, so I—yeah, I imagine this batch is gonna go even faster.
Josh: I got my, like, whatever clicker finger ready to refresh the page.
Ben: Well, Josh—what Josh is not saying is that he was kind enough to buy my ticket to Rails World for me during that small window because I was on a plane, coming home from a trip and I would have missed it if not for Josh's kindness. So now we gotta try and make sure that he can also participate.
Josh: That’s the kind of guy I am.
Ben: That's true. I knew he wouldn't brag about it himself so I had to give him the props.
Josh: Well, thank you.
Ben: So Adam, tell us about Judoscale. I'm sure there are probably one or two people who might not know what it is. So let's talk about that.
Adam: Yeah, JudoScale is an autoscaler for Heroku and other cloud platforms, such as Render and AWS, but the reality is mostly Heroku. So, for seven years it's been a Heroku add-on, and just last year we built an integration to Render and AWS.
But, yeah, if the native autoscaler on those platforms don't meet your needs, Judoscale steps it up a notch and lets you do things like auto scale your Sidekiq workers and auto scale your web stuff based on request queue time rather than response time. So, it's just a better auto scaler. That's all it is.
Ben: I mean, there are worse ways to make a business, than to build a better something than someone else has already built, right? That's a pretty well worn path of a good product.
Adam: Yeah, it has worked. It has worked pretty well.
Josh: It is kind of cool how Heroku has built like some simplified features that they allow add-ons to kind of, you know, do the more advanced version.
Adam: Yeah. Heroku add on marketplace has just been fantastic for—it was—Judoscale was a side project for the first five years of its existence. And the marketplace let me just throw it in there and not really market it and let it just grow slowly on its own, which was super cool.
I was actually thinking, before hopping on here, that, like, Judo scale, like I started building it in 2016, launched it in 2017. How old is Honeybadger? You guys are a little older than that, right?
Josh: Yeah, we were 2012. Yeah.
Adam: You’ve been around a long time.
Josh: A long time.
Ben: I was talking to a friend of mine at church on Sunday. And, we—he was just—we were just catching up. And he was asking me about Honeybadger. And I was like, “Yeah, we've been doing it for 12 years and et cetera”. And he was talking about that and he's like, “Man, I've been married for 12 years.” That’s—so for him, that was an interesting kind of perspective. I was like, “Yeah, that's kind of weird”. Yeah.
Adam: Yeah, you and Josh have essentially been married for 12 years
Ben: Well, longer, like we've been working together for, I don't know, what is it like 15, 16 years now.
Josh: We've been working together longer than I've been married to my wife, yeah. We're 14 years, this year.
Adam: And you guys are a team of four now. I was looking at the site before this. The meet the Badgers page has four of you. I think.
Ben: Yeah, there are four people on the about page, but we have one hidden employee, because he was just recently added, and we haven't gotten around to putting him on the page. It’s my son—actually has been working for us lately, that's been fun. At first, I—actually, to be honest, I wasn't quite sure how it was going to go, because working for your dad can be kind of, you know, can be a weird experience. But it's been fantastic.
I love having him work for us and he's done some great work and he's going to be editing this podcast audio. So, he's going to be hearing me talk about how awesome he is. So, go Addison. But yeah, that’s—so, we got to update our about page and get that, so we're five people
Josh: Yeah. We should do that. I've also been really enjoying having Addison around and he’s been super helpful with everything. That Honeybadger banner went off real—it was perfect at the conference. So nice work, Addison. We had him do a, like, roll up banner for, whatever, you know, next to the conference entrance.
Adam: At which one, at Rails Conf?
Josh: That was at Blue Ridge. Yeah, so I actually had to travel with like this—whatever the banner comes in, like this carrying case and I had to like, check it. And, you know, everyone thought it was like a MIDI keyboard or something. Unfortunately it wasn't that cool.
Ben: That would be cool to have the conferences though, you know, have—so we do a sponsorship and we have a booth and you're sitting there doing some tunes on the keyboard.
Josh: I should do that. Yeah, I should. That would be pretty fun actually.
Adam: You guys always have just like really creative conference, not just conferences, like. every part of the Honeybadger brand and stuff is always just—like creativity is just kind of oozes from—who is the creative genius, or is it both of you, or is it all of you?
Josh: I don't know. We all have some pretty good ideas and then I think like it all synthesizes and then—I’m usually the one who executes. I handle most of our marketing these days, although I got to like give credit to in the past, to Starr and Ben Findley when he was here. Definitely been a team effort, but yeah, I definitely like the weird creative projects.
Adam: Who does all the—you guys have a lot of custom artwork for, like, your swag, for—I mean I'm thinking like the exceptional creatures stuff, and just all—all throughout the website and stuff. Do you—does somebody on the team an artist, or is that stuff that you contract out?
Josh: So a lot of our artwork, on the website and on our t shirts that you've seen in the past is by an artist called Kyle. His name's Kyle Shold and he hasn't been doing our art lately, but yeah, he's a really good artist. He actually—I met him locally up here in the Pacific Northwest.
He's done a lot of the art for, like, a lot of the breweries, like local breweries around here, like their labels for their beers and things. So, love his style. And then, the exceptional creatures was actually illustrated by my wife, Caitlin. She's an artist and yeah, we worked on it together.
Actually I have a few—I think I have a few creatures I need to add to the page. Didn't mean to get back around to that, but yeah, that was a lot of fun, and then I've got a new t shirt design. I'm hopefully working on with a cool artist. I've actually—so he’s—I've been telling a few people this story, but there's this artist that has done—so there's this company, a skate company Santa Cruz Skateboards.
Some people might've heard of that, he is out of Santa Cruz, California. And they're like, one of the old school, like, really well known skate companies from when skateboarding kind of blew up in the seventies, I think. And their art is, like, really well known cause they've had the same artists for years and actually it's now passed on to his son who's been doing their art, since, I don't know, for the last 1—maybe, I'm not sure how long, over 10 years, but I like randomly emailed him and he's gonna work on some art for us. So I'm pretty excited about that.
Adam: Yeah, that's awesome.
Josh: But yeah, like just, sorry, go ahead.
Adam: I'm gonna try to take some inspiration from you guys. And Judoscale needs a little more creativity behind our marketing efforts. We haven't actually had swag in like—haven’t even made t shirts since we rebranded from Rails auto scale to Judoscale, which at this point has been over two years, probably almost three years at this point, mostly just because I haven't really liked our logo.
And so, I just recently had worked with my brother to redesign our logo. Gonna start plastering that on some things and actually have some swag. It was very sad to go to—like, to go to RailsConf and not even be able to wear a shirt for my own company, let alone have some to give away or anything like that.
Josh: Yeah, I have one of your shirts from back before, somewhere. I remember I was at a Rails Conf, trying to remember what one it was. It was like, I don't know might've been—I can't remember if it was before the pandemic or not. It's like a blur, but yeah, I got a shirt from you at one point, yeah, So that's cool.
Yeah. I—like, my only thing, my thing with shirts is just like, just make a good one. Pick good material and I like—a nice design is a good thing as well. But just, like, when I see companies like, you know, buy the cheap shirts that don't fit well, I'm just like, “why do that?” Because if people don't want to wear, you know, if people don't want to wear it, like what's the point of making it. I remember yours was really well done.
Adam: Yeah. Thank you. I appreciate that. It—that—I remember that conference. I brought a whole bunch of shirts with me, just to give away, like I wasn't a sponsor or anything. I didn't have the money to be a sponsor, but I just had, like, a suitcase of shirts and was just trying to, like, give them away and that was a huge pain because you gotta bring assortment of sizes. So, here's my idea. I'm just gonna, like, throw out an idea and see what you guys think. For RailsWorld, I just wanna have, like, a QR code that anybody can scan, and then, I'm just gonna take them up to a page, just like, just put in your address, I'm gonna send you a shirt, but I'm not bringing a bunch of shirts to the conference.
Ben: So we've done that. We've done that. We use PrintFection for a lot of our swag fulfillment. And we, basically, what they do is you can send them or have them print a bunch of inventory and a variety of sizes and whatever. And then you can have individual redemption links which you can generate via their API.
What we do is we actually have in our little admin tool that we built for ourselves, we have a page where we can call that API, and we have a QR code generator on that page. So that when we're at a conference, we can open up our phone, we can generate a new link, show somebody that QR code, and then they can hit that and go to the Print Infection page to get—enter in their address and get it filled for them.
Because yeah, we had the same problems. Like, you show up at a conference, you got the suitcase for one—you got to get the suitcase of shirts there. That's a pain in the butt. You got to drag that through the airport. And sometimes it gets left in a cab as Josh well knows.
Josh: Or at Bag Check, uh, you left at Bag Check and you drive like 30 minutes to Pittsburgh and then have to come back, you have to drive all the way back in rush hour traffic.
Ben: And then, and then once you get it there, you got to hand them out and you probably run out of the larges or the extra larges before you run out of the smalls and the mediums, you know, and then you don't have any to give out to people that want them. And so, yeah, the QR code has worked out pretty well for us.
The downside is you have individual shipping charges now versus, you know, just bring it in the box. But all things considered, I think it—we decided it's worth it to pay the extra shipping costs, yeah.
Adam: Yeah, I feel like, I mean, yeah, on the flip side, if you make a bunch of shirts, like, I still have probably 50 rails auto scale shirts, there are basically just rags now, like a lot of rags, never going to run out of rags at home. It's got all these shirts that I don't know what to do with.
So, yeah, I think I'd rather just like, pay extra for each individual shirt to ship them individually, than just have to deal with inventory. Just having all these shirts around.
Josh: Yeah. Yeah, it’s—I highly recommend it if you can make that happen.
Ben: It's also enabled us to have that as a giveaway promotion sometimes in the app. So we've done at various times, like, “Hey, if you enter your payment information before your trial is done, then you can get this shirt”. And then we just respond. So they ask them—we ask them to respond to us and let us know they did that.
And then we send them an email with a link that they can then get the fulfillment done for themselves. So, it’s pretty handy. It's not the cheapest thing in the world, but it's fun to do things like that. And people like that—that stuff.
Josh: Yeah, we actually have a shirt offer right now still going for people who send production data to Honeybadger Insights, our new logging observability tool. And so you can go to Honeybadger-dot-io forward slash two, then like the number two legit. And that we have a little landing page for that shirt.
And it's a fun, like, Honeybadger and a elephant. The elephant is supposed to be like, it was when we first did that design, we’re trying to do like a Honeybadger, PHP, like, Laravel sort of thing. So we did that like a PHP elephant character. Kyle did that for us. But, it’s—yeah, it's a fun limited edition, Honeybadger shirt.
Ben: So yeah, highly recommend the not bringing a whole box of shirts, but you know, if you've got 50 shirts left over, maybe—I guess you can take them down to Goodwill or something. Somebody will pick those up and have some shirts.
Adam: They're going to have some collector's items
Ben: Yup. So I—I’m interested, you mentioned you moved—you had, you started Heroku and then you added ECS, which we use here at Honeybadger quite a bit. So I'm familiar with that, , but we don't use Render and I was kind of interested in that, since I'm not familiar with it as much. How does the autoscaling with render work compared to the others? Is it pretty much the same kind of thing?
Adam: Yeah, it's pretty much the same kind of thing. On basically any of these platforms, we have an integration library that lives in your app. So, if you've got a Rails app, it's a gem that you install, so that we can measure request/queue time, how requests are queuing up in Puma, which is basically a measure of how at capacity you are.
We measure that, report it up to JudoScale, and so that just works the same whether you're on Heroku, whether you're on—whether you're on Heroku, whether you're on Render, whether you're on ECS. Then the only difference is on the JudoScale server side, you know, are we scaling up Heroku dynos, Render instances, or Amazon ECS tasks? But, it basically works the same no matter what platform you're on.
Ben: That's cool.
Adam: The only difference is, like, with Heroku were able to automate a lot of the installation steps because you're installing it through the Heroku marketplace. So when you install the Heroku add on, we get some API access to be able to scale your application, whereas on Render, there's not like a marketplace.
So it's more of a, you sign up through Judoscale-dot-com and then we give you an environment variable that you need to set within your app, and you need to give us API access. So, there's just a little bit more set up that needs to happen. But at the end of the day, it all works the same.
Ben: And it'd be like the same story with ECS too, right? You have to give privileges to the account to be able to call the APIs for you and that sort of thing.
Adam: . Yeah, had to do that IAM dance and all that stuff. Yeah, which I mean, that was the hardest part for us because I don't have much experience with AWS. So building that integration just, you know, required a lot of getting familiar with how all that—all the permission stuff works.
Ben: Well, I have to give you props on your gem. Speaking of monitoring Puma and spinning up things based on those statistics, so, we recently have undergone a dev cycle where we're just wrapping up where we've added a bunch of instrumentation, automatic instrumentation to our gem that we can report to Insights.
So the goal being, you add Honeybadger Gem or upgrade your Honeybadger Gem to the latest version. And now you enable one feature and you're going to send a bunch of stats to Insights. Like. Puma backlog and thing request latency and that sort of thing. And, , so what we did is we, , made a list of all the gems, maybe, , probably not all, but a good chunk of the gems that do this kind of monitoring and yours was in the list.
And, yeah, got compliments from Roel who did this work on your gem, in particular, on how clean it was, nice it was to look at and to get those kinds of statistics out there.
Adam: Oh, that's awesome. That is mostly Carlos's work from a couple years ago. He did, like, a short contract with me just working on the gem stuff. So yeah, I'll pass that along to him. But it's definitely—I’ve been really happy with how that has turned out. It's been easy to work with.
Josh: Nice job, Carlos.
Adam: Yeah, Unlike our other integration libraries. I just—I‘ve spent like the past two or three weeks working on our Node integration. It’s just kind of amazed me how difficult it's been for me because I don't think I'm a bad programer—I think I'm okay. And I know JavaScript, but I don't really know the Node, like, ecosystem and all this stuff.
So it's just been one thing after another. Just in terms of getting things packaged correctly, and getting things set up correctly, and do I need a build process, do I not need a build process, just all these things that have been not fun to deal with. I'm ready to be back on Ruby.
Josh: Yeah. We feel that For sure. I think like out of our, if you like measure the whatever, put it—have a pie chart of the work we put into our client libraries, like JavaScript is probably like, I don't know, like half or, you know—
Ben: Or more.
Josh: Yeah. Yeah, or more.
Ben: Yeah, because there's so many flavors and we get—we get support requests from every kind of combination you can imagine. It's like, “Oh, I'm running this NuxTap, but I'm doing this other build tool and I've got this background thing”. And it's like, “What?” You know, it's like, I haven't heard of of those things.
Adam: Ben, I remember you saying you've been actually enjoying writing some Go lately. So I take it you haven't had those kinds of—or maybe you've just worked through them. But like, has it been a struggle getting familiar with the go ecosystem or have you—or has it—have you been able to pick it up pretty quickly?
Ben: I wouldn't say it's a struggle because there's typically, you know, one approved way to do a thing, right? Or, maybe two. There's not 50 gajillion frameworks in Go for any given task, right? It's like, well, there—there are a number of routers, like web routers, that's interesting. There's, I don't know, five or six probably popular ones.
But in most cases in Go, it's like, “Oh, just use this library. Everybody uses this thing, so just use that thing”. Or do it this way, right? Versus, whatever is the flavor of the month kind of thing. So, that’s—it's more straightforward.
Josh: And the Go standard library also has like it—it’s very complete. It has something for most things that you would want to do, even if there are, like, alternative packages now, I just remember when I learned it—like it was a while now, a while ago, but I was really impressed that it had like a lot of official things built in, like, web servers and things like that.
Ben: Yeah. And then in the later versions of Go, like I was just reading yesterday, like, version 1. 21, I think they just brought in a new standard library for structured logging slog. There's been zero log, which a lot of people use, and there've been a couple of others. And so it seems like they're on this happy path of like, okay, we—kind of like Rails does sometimes where you see gems that end up in Rails, like after it's been, you know, a certain amount of popularity and then go just like, “Okay, now it's in a standard library, so just use that”, right? Don't worry about all those other ones that are out there.
Josh: Yeah. I really like that approach of having all of that officially maintained—
Ben: Yeah.
Josh: and in the language. It's pretty cool.
Ben: Yeah, I think we could use a bit more of that in the Rails world, to be honest. I like that part of the Laravel world where it's like, oh, you know, there's a difference there because it’s typically one corporate entity that's behind it. But it is nice to have, like, this is the thing that you do, right?
As opposed to, these are the six different options you could choose from. But I think really, the pain that I have with JavaScript is just the churn. It's like, “Oh, you know, this month, this is the best way”. And, you know, six months later, “Oh no, that's crap. This is the best way”. So at least stick with something for a while that would be nicer.
Josh: I think you really nailed it. Just like with the everyone's app, everyone has a different framework that, like, that they're using. If you like have all the different combinations, I'm sure you could do some math to like, figure out like, how many combinations given all the popular frameworks that are out there.
But Adam, you don't have to do any client side. You said it's more Node you're dealing with. Just be happy that you don't have client side JavaScript to deal with on top of this, because we had to like—we have a universal package, which has to work in the browser and in Node, because some people do both at the same time, you know, that you got like server rendering React for the client side, there has to be like, parity on either end and it's, yeah, it’s interesting.
Adam: That sounds horrible. Yeah, I’m glad I am thankful that I've only had to deal with the server side JavaScript for this.
Ben: So you have a Ruby support, you have JavaScript support. Are there other platforms that you support as well?
Adam: Uh, we support Python as well and I did not—well, I wrote the first version of that and it was pretty gnarly, so I ended up finding a Python developer to rewrite it. Somebody who actually knows Python and knows the ecosystem and he did a great job.
And he's actually still helping out a little, you know, a few hours each month, just, like, handling customer support issues that are specifically dealing with Python because I have limited knowledge there. And yeah, if somebody—so one—one of the, Python background, like, job queuing systems is called Celery. So like, we've had a few customer support tickets dealing with Celery recently. It's like, “Oh, my cues aren't showing up”. And I'm just like, “Man, I have no idea.”
Ben: Yep.
Adam: I'm very happy that I have somebody else. Like, please look into this. Because this is too much.
Josh: That was our—that’s been our approach for the client libraries for the most part. I mean, like, we built a lot of the—a lot of them, like you in the earlier days, but having a contractor for each one that subject matter expert works. That's a pretty good fit for contractors, we found.
Adam: Yeah. So how many do you guys have at this point? Not contractors, but platforms
Josh: Yeah, I don't know
Ben: We've lost count. That's quite, yeah, that's quite a few. And we've been—so we got, let's see, well, we got the basics. We got Ruby, Python, JavaScript, Go, we have Elixir, we have—I think someone did Clojure. I think someone might have done Haskell. I know someone did dot NET. That's all I can think of off the top of my head.
Josh: And then don't forget like we have React Native and Coco and a few native ones as well, Java.
Ben: Yeah. So yeah, and the having the contractors has been a lifesaver because like, I don't know anything about Java, right? And I know very little about Cocoa. So having contractors who actually know that stuff has been super helpful. One problem that we encountered, that Josh has done a really good job dealing with, is like, you know, you get a contractor who's working with you for a few months and then you're not giving them a full time job, so they go find something else to do.
And maybe after a while, they don't have any time to work with you anymore. And so you got to bring someone else up to speed. It's like, okay, we've got to replace that person. And so Josh has done a really good job of documenting our processes and how does our open source approach work and how do you get stuff into a client library, regardless of which one you're working on.
And that's been super helpful in getting new contractors spun up pretty quickly. And so, now we can basically go to Upwork and we can be like, Hey, I need a whatever, person that knows Python, like our current Python contractor came from Upwork.
Josh: Yeah. And it works well with—even if it's someone on Twitter or Mastodon and it's like, “Hey, I got some time”, and you're like, “Hey, well we have some, whatever, Elixir work available”. We have—we use sign well for signing that contract. So we just have a contract ready to go.
You just sign the contract. And then we have a whole, just like basically just a short checklist onboarding process to get someone in. And then the documentation to , we have them read through it. Learn , how we do things. And then hopefully there's issues waiting for them to work on. So it's a—it’s pretty streamlined, even if it's not on a platform like Upwork. But we've had pretty good luck with Upwork lately as well.
Adam: Yeah, definitely an area that I need to improve on is getting better about documenting processes within the business. And, like I said, yeah, Carlos is my first full time hire and I've had a few contractors work on a few things, but definitely don't have that sort of repeatable process ironed out like that, which sounds pretty great.
Josh: I'm curious about how the rebrand went, that’s not something that everyone does. You said it was a—it’s been a couple of years since you rebranded as Judoscale from Rails Autoscale.
Adam: Yeah. Obviously when you start a product called Rails Autoscale—makes it hard to then branch beyond Rails. But I wanted to—obviously there's a lot more developers out there than just Rails developers, and JudoScale could, well, the autoscaler could be useful for all of them, Knew we wanted to branch out beyond Rails, so I thought, we need a new name because Rails Autoscale just doesn't make sense.
The hard part about rebranding was that as a Heroku add on—you can't rename a Heroku add on. We couldn't just say, “Oh, the Rails Autoscale Heroku add on is now JudoScale. So we actually had to spin up a new Heroku add on, which meant we had to go through the barriers of when you want to create a Heroku add on and get it into the marketplace. You have to get 15 customers before you're even in the marketplace, and even then, you've got a beta flag and you can't charge anybody until you get 100 more customers.
So 115 people need to install your add on before you can even start charging money for it. We had to go through all of that a second time with JudoScale. So, that was kind of painful. Other than that, it went pretty well. We do still have both add ons in the Marketplace because I didn't want to just like shut down Rails Autoscale.
We still have hundreds of customers using it, so it's a little weird that we have two add ons in the Heroku Marketplace. Different names, but they’re—if you install one, it's the same backend app. It's everything. It gets exactly the same—there's one database, no matter which ad on you install, there's one app running in the backend.
Josh: Interesting.
Adam: Yeah. For the most part, it hasn't resulted in much customer confusion, I don't think. I still haven't heard a lot about it.
Josh: Can you still install the Rails Autoscale version or is it just for existing?
Adam: Yeah
Josh: Okay. So like, what's the—has Judoscale, I assume Judoscale surpassed it in the marketplace? If like, how do people—like when people search for auto scaling, I'm wondering if is there any confusion there?
Adam: If you search in the marketplace, there's actually—I think Heroku sorts it randomly, honestly. Like I don't, yeah, there's nothing you can do to boost yourself in the rankings or anything like that. And it just—I think it is like a random ordering.
Josh: Okay, yeah.
Adam: So yeah. People do still install Rails Autoscale. On the marketplace page for Rails Autoscale, it does say Rails Autoscale is now JudoScale. You can go here. But I didn't want to put anything on there. Like, don't install this or don’t—or, you know, I could have even disabled all the plans so that you couldn't install it. I don't want to do anything that's going to potentially have somebody land there and be like, “Well, I'm not, I can't use this thing. So I'm gonna go find something else.” So if they want to install it as Rails Autoscale, they still can.
Josh: Yeah.
Adam: Unfortunately, even like, even in Google, Rails Autoscale still, outranks Judo Scale for a lot of things, but—
Josh: Yeah.
Adam: Yeah, at the end of the day, it doesn't matter that much to us which one they sign up for.
Josh: Yeah. I'm surprised Heroku doesn't have a way to just migrate or, like, you know, rebrand an existing add on, or something like that. That seems like a lot of extra work you had to do.
Adam: Yeah, was surprised too. But, you know.
Ben: It really feels like—having worked with their add on stuff, it really feels like it's a rotisserie chicken, you know, they just set it and forget it. It feels like they were—they built it and they got it to a certain point and it works and now they haven’t looked at it again in like eight years or whatever.
Adam: I mean, it kind of felt that way for a lot of parts of Heroku for a long time. Really, just in the last year or two, it seems like they’ve started taking it seriously and making improvements and, even hiring. I'm excited to see the things that they're doing—that they are actually doing things now.
Josh: That's, yeah, that's great to hear. I love Heroku and I am glad they're making it better.
Ben: I'm, happy to hear you say that after having read your blog post about the experiment with the Router 2 point 0.
Adam: I'll give a—I’ll give a quick summary there. So, Heroku has a new router, which supports HTTP 2 point 0 amongst other things. And it's in beta right now. And so, you can try the router. And so, we were like, “Yeah, let’s try the new router”. We swapped it over in our staging app and it worked fine.
So we swapped it over in production and our app just completely fell apart. But we're like, all right, that's all right, a few minutes, you know, latency just started climbing. No matter how many dynos we ran, they're like, I don't know what's going on. Let's just switch back to the old router.
Switched back to the old router, and it got even worse. And after about a half hour, we're like, I don't know what else to do to salvage our app, no matter how many dynos we run, like, we're just getting timeouts and the actual time spent in Rails was still very short, like 10, 20 milliseconds, really, really quick responses.
But our overall response time was still terrible and there seemed to be nothing we could do to bring it down. So we ended up completely rebuilding our production app on Heroku, spin up, like, Heroku Create. And Judo Skill Prod 2 is now our production app on Heroku.
Josh: Wow.
Adam: So yeah, uh, that was not fun, and I'm still working with Heroku to get to the bottom of what happened there. I think they did find a bug. internally for how connections are held onto when you switch routers. And that seems to be part of the problem. But yeah, still working with their networking team to figure out what happened there.
Josh: Yeah. That sounds like an ordeal.
Ben: Yeah.
Adam: Not a fun way to spend a Saturday morning, but I mean, I brought it upon myself. Like we didn't have to try the new fancy thing. We really didn't need it. I just thought Heroku has a new thing. I want to try it. We also did not load test our staging environment with the new router. There's definitely some of the responsibility falls on us.
Ben: Yeah, that's one of those “This is going to end the business!” kind of hair on fire kind of moments where you're just like,"Aah, take a deep breath. We'll get through this.”
Adam: Yeah. I was thankful. I was thankful to have Carlos, because he actually—he happened to see like all the alerts piling into slack and he's like, “Uh, something going on. Do you need any help?” I'm like, “Yeah, hop on tuba with me, please. I don't wanna do this alone.”
Josh: Oh man. Yeah. That must've been so nice to have someone. I just can't imagine like dealing with that as a solo just yet, just by yourself.
Adam: I mean, , that's the way it's been for seven years. And anytime something went wrong, it was just me on my own.
Josh: That just makes me thankful for my co founder. Yeah.
Ben: Yeah, it is definitely nice to be able to swap off some of that responsibility and share that load. It helps a lot for the mental state. Well, Josh, I guess we got to get you out of here so you can get ready to click that mouse, refresh that webpage.
Josh: Got about fifteen minutes until Rails World launch. So we'll see, cross your fingers for me.
Ben: So oftentimes, Adam, we share like what we're, what we've been doing recently or what we're going to be doing next. We talked about how we have the gem update coming real soon. And we also have some Rails dashboards coming out, on top of that instrumentation. Basically, if you start seeing that instrumentation data, we'll show you a new dashboard—the Rails dashboard, that has pretty charts automatically done for you.
And we're looking forward to having that out there. We're using it ourselves and it's pretty cool. And then, doing that some other platforms as well, but yeah, Adam, do you have anything cooking up that you want to talk about?
Adam: We're just getting started on a new project to make some big changes for how we handle log based auto scaling. So, if you're using, like, a language that we don't have an integration library for, like I said, we have Node, Python, and Ruby, but if you're using Java, if you're using PHP, Go, anything like that, we don't have an integration library, but you can still autoscale your app based on response time, which we just get from Heroku's router logs. But we're going to be diving into getting some more data out of those logs so that it's useful, like, as soon as you install it, you don't have to install an integration library right away. It's just going to be, like, immediately you start seeing some useful data and you can start autoscaling. And then If we have an integration library available for your stack, we'll offer it, and you can use that, but just trying to make Judoscale more useful for everybody rather than just the select few that we have an integration adapter for.
Ben: Nice. Yeah, there's a lot of data in those router logs. We've spent some time on that for when we built out Insights and pricing that data. There’s—and like some of the add ons will put metrics back in there like Redis and PostGres will dump some metrics in there so you can keep an eye on those things and not have to have an actual agent running. So that's cool.
Josh: Maybe we'll have to have a Honeybadger-Judoscale, integration of some kind in the future. It'd be cool. Could scale based on your insights, blog stream or something.
Ben: I know from being a JudoScale customer with one of my clients that JudoScale does put those kind of events back into the logs that you can pick up in the Heroku logs. So yeah, we could definitely pull that into Insights so you can know when scalings are happening.
Josh: We could have, like, an auto scaling dashboard. Yeah, that'd be cool.
Ben: Let's get on that.
Adam: Love it.
Ben: Well, cool, Adam. This has been fun to chat. It's been fun to hang out with you.
Adam: Thanks for having me, guys. This has been great.
Josh: Yeah. Ben and I have been really enjoying talking to founders like Adam and John, and we're hoping to have some more friends of the pod on in the future. One thing that listeners can do to help boost our rankings and get more people listening is, go give us a review in your podcast platform. And otherwise, you can find us at founderquestpodcast-dot-com
Ben: See ya.