Show Notes:
Links:
John Nunemaker WebsiteJohn Nunemaker TwitterFlipper CloudSpeaker DeckScientist
Full Transcript:Ben:
So today we have a special episode of FounderQuest. We have John Nunemaker with us, instead of Starr. Starr was taking a break today. And Josh and I are going to be chatting with John, and talking about the fun things that John's doing. John, I got to start off by saying that I'm a huge fan. I've been following your work since the Harmony days, back at Ordered List, I guess that was ... I don't know when that was, 2000 and something.
John:
'07, 8, yep.
Ben:
Yeah. Yeah. So I think I got introduced to you through the Rails community, being back in the early group. So I don't remember how exactly we bumped into each other back then. But I remember Harmony was pretty cool, and the other stuff you did with Ordered List.
Josh:
Yeah.
Josh:
I was newer to the Rails world back then. So both of you are Ruby celebrities to me. So yeah, it's cool. It's cool to have you here.
John:
Thanks so much. Yeah, I'm really excited about it. I feel the same way about you guys. Especially I remember I was at RailsKits.
Ben:
Yep.
John:
I remember ... Yeah.
Josh:
Yeah.
John:
I remember that. I remember a bunch of the stuff back in the day. And is it Stympy, or something? Website?
Ben:
Yeah.
John:
Yep.
Ben:
Yeah.
John:
Oh, yeah. That stuff sticks out.
Josh:
Nice.
Ben:
Nice. Nice, cool. Fanboys all around. It's awesome. And you're a prolific open-source author. We have in fact two of your gems in our app right now. We have nunes, and we have httparty running in our app. So thank you for those.
John:
That's awesome.
Josh:
Yeah.
John:
That's really cool.
Ben:
Yeah. I love nunes. And I love the description of it. It's like, "This is the monitoring app I would add to your app if I was working with you."
John:
Yeah. I feel like stuff like that, I get lucky and it sticks. But it's just this moment where I'm like, "I got to come up with some kind of a description. I really don't want to do this. What should I put?" And then it's like, "This is what I would. I would do this if I were you." So I'm just going to put that as my description and peel out.
Ben:
It's cool. But I think the ... We're not going to talk about this much today. But I just wanted to toss this in here. And I think one of the projects that you've done that I'm most interested is probably one there is least information out there about. And that's Haystack at GitHub.
John:
Yeah. Yeah. Hey, I can answer any questions related to that too. On air, off air, whatever you want. Yeah.
Ben:
Awesome.
John:
Yeah. I worked on that for a little while. I didn't build it, but I tuned it a bunch.
Josh:
Remind me what Haystack does.
John:
It was the exception tracker-
Josh:
I remember now. Yeah. Cool.
John:
Yeah. Yep.
Josh:
I have built a few of those.
John:
Yeah, I don't know if you guys have heard of exceptions.
Ben:
Yeah, we did a little bit in that line. But yeah, I remember reading some of your blog posts about Haystack, and I was kind of jealous. I was like, "Oh, man. It'd be cool if we got GitHub as a customer." But yeah, I totally understand why you'd have something totally internal and custom to what you do there at GitHub.
John:
Yeah. I still always wonder if they still have ... I need to reach out to people who are still there and ask. I'm always curious what technology has lasted and what hasn't, and stuff like that.
Josh:
Yeah.
Ben:
Yeah. So how long have you been gone from GitHub?
John:
I would say ... Hard to remember. I would say 2018 I think is when I left. So it was right when after the Microsoft stuff went through. And it happened to coincide with paternity leave ending for me, and all the ... Just perfect timing. So all the stuff kind of came down at the same time. And so my last day of paternity leave was a Friday. And that Friday was the day they closed the deal. And then that Monday, I resigned and moved on to the next stuff. I love GitHub. You can see behind me.
Ben:
Yeah.
Josh:
Yeah.
John:
No one listening can, but I have an Octocat behind me in my room. It's completely office is stuffed with Octocats. I'm a huge fan still. I just am not a big company person really.
Ben:
Yeah. Totally can relate to that.
Josh:
Yep.
Ben:
I've never thrived in big companies. So yeah, getting acquired by Microsoft would make GitHub a pretty big company.
John:
Yeah. And it was ... I mean, we were 45 through 50, and then watched it grow over six, seven years to in the thousands.
Josh:
Wow.
John:
And it was just totally different than we had started. So it was-
Ben:
No doubt.
John:
Yeah. And that's kind of where Flipper and Flipper Cloud and stuff like that even came from was because I was working there. And not to jump ahead or anything like that, but that's ... I was like, "I know I'm not going to be a big company person. So I got to come up with some kind of a runway, because I'm the guy who runs off the clock in the fourth quarter." I'm very safe and conservative in my moves. So yeah.
Ben:
I love that. So let's talk about that. That's very interesting.
Josh:
So you're in good company. That sounds a lot like Ben.
Ben:
Yep.
John:
Yeah. So basically it was like ... I mean, Flipper itself, I started in 2013 just for fun on the weekend, which was a lot of ... Httparty, a lot of gems like that, that's where they came from, was just hacking around on the weekend or in the evenings. I spent a lot of time doing that kind of stuff. And I have always been interested in feature flags, because I worked on, a long time ago. I don't know if you guys know this or not. But I worked on Words With Friends, the Scrabble game on the backend. So I didn't work on any iOS stuff, but worked on the backend. And every time I tried to roll out, I always joked that that time period in my life, all I did was write caching for a year. Because it was just trying to ... We scaled from 50,000 requests a minute to over a million. It was insane.
John:
And so we were just trying to keep the service up. And that's where feature flags came in, and it kept going down every time I tried to roll out this new caching. And the new caching was really important, and I couldn't get it to roll out, because every time I added it, the whole site would just screech to a halt due to a cold cache. So that's when ... I was working with Jesse Newland I don't know if you guys remember him from Rails.
Josh:
Oh, yeah.
John:
Yeah. So we were working together on it. We were like, "We should do feature flags." He was like, "Check out this thing called Rollout." And so I set it up and got it working. And we slowly rolled it out. And then I was just like, "Wow. This is amazing." So yeah. So then a couple years later though, I didn't love the API. I'm real picky about APIs and the way the code looks, and the way it feels. And their examples used like, dollar rollout equals, or something. And dollar just made scrunch my shoulders and nose, and everything.
John:
So at that point I was like, "I think I can do it better." And that I feel like how I always end up in open-source is some kind of silly idea like that. It's usually like you change one thing, and then everyone's like, "Why did you make a new project? This is just one small change." But that's how it started. And it didn't get used by me for I don't know how many years. It didn't get used at GitHub even. So I was at GitHub at that time. And it was probably 2015 actually when my son was born, and I was on paternity leave with him. That's when it got added to GitHub. Two guys, Adam Roben and Rob Sanheim added it. And so they were kind of working with me a little bit. But I was obviously kind of off the grid at that point, struggling to find out how to be a parent.
John:
But yeah, that's kind of how it came about was then, I just kind of played on the weekend and thought it was fun. And a bunch of people started using it. I knew I just wanted to make it not tied to Redis. Because I've had interactions there that weren't fantastic. And I liked the idea in storing it everywhere. And so that's kind of ... I was like, "Let's just make something that can talk to anything." And that time, Rollout didn't. Now they have more of an adapter idea in theirs as well. But yeah, so that's kind of how it started. And then eventually those guys pulled it into GitHub, and I helped tune it a little bit, and help with that. And then from there, it just kept pulling my interest back. And so I just kept kind of playing with it.
Ben:
It's cool. Yeah, we've used Rollout a little bit in Honeybadger. We don't do feature flags very often. We typically deploy ... If there's something that we feel is kind of dangerous, we will deploy it to our internal instance of Honeybadger first. And so we'll inflict it upon ourselves, and make sure that it's stable and stuff before we unleash it. But every now and then we do have a feature flag. And yeah, Rollout's API, I think we're still using the old-fashioned, activate user, and stuff.
Josh:
Yeah.
Ben:
And yeah, it's-
John:
Yeah.
Josh:
Maybe that's why we haven't used it as much.
Ben:
Maybe.
Josh:
We're afraid to interact with a global variable.
Ben:
Exactly. Yeah, and now we have Standard running all the time on our ... To lint our code, right? And so now even in VSCode, you can it's like, "Oh, don't use that dollar sign."
Josh:
Yeah.
Ben:
" ... man."
John:
Yeah, and every other instance of life, a dollar sign is good. It's just in Ruby where you're like, "Oh, this makes me feel bad."
Ben:
Yeah, yeah. But then again, I came to Ruby from Pearl. So sometimes it's like, "Ah, it's a kind of throwback." It makes you feel warm and fuzzy to have all those siggles.
John:
Yeah.
Josh:
So it's probably not great if your API is a linter error.
Ben:
Yeah. It's probably bad news.
John:
That's neat that you guys have an internal instance. So you use that for Honeybadger itself then?
Ben:
Mm-hmm (affirmative).
Josh:
Mm-hmm (affirmative).
John:
Oh, okay. Cool.
Josh:
Yeah.
John:
I mean, it makes sense. I just hadn't really thought about it.
Ben:
Yeah.
Josh:
It's pretty handy. It's kind of like a production staging instance. It's live, taking a lot of traffic like live production traffic. But yeah, we can kind of play around with it.
John:
We did that at Words With Friends. We had, I think it was called a smoke instance. And it's the same idea as a canary or something. It was kind of pre feature flags. It was like we just had one thing that got 5% of the traffic. And so we could roll out to that first. And if it went well, then we would tag it and roll it out to product and stuff.
Josh:
Oh, yeah. That's a good idea.
John:
Which is not the same. But kind of the same idea.
Ben:
Yeah.
John:
That's cool.
Ben:
Yeah. One of the benefits has been, now we have ... So it's deployed in a separate region. So we're a 100% AWS. And so our internal instance is in a different region, so it's obviously a different VPC and everything. It's just completely separate from our production stuff. And we're big fans of Ansible and Terraform. And so the nice side effect has been, now we have this good set of scripts in Terraform and Ansible to spin up an entire stack pretty easily. So it's been a lot of fun to play with, kind of geek out on that sort of thing.
John:
Yeah, that's awesome.
Ben:
So you said Runway. That was a trigger word for me. Because yeah, it's like Josh said, I'm totally into having backup plans and that sort of thing. So at some point, Flipper went from this fun open-source project to, "Maybe I'm going to do something else with it." So tell us about that.
John:
Yeah. In 2017, I was ... I think it was 2017. But GitHub had just exploded growth wise, and stuff like that. And I was just kind of feeling the big company. At a big company, and there's nothing wrong with this. This is just kind of the way it is. But I feel like you spend a quarter of your time just telling everyone else what you're doing with the other portion of your time. And that was just starting to wear on me. I just like to get in and make an impact. And it's difficult to make an impact that you feel is substantial at a company that's thousands of people.
John:
And I think part of it's just a mental issue. Because I see other people who definitely make an impact. And I think they're just maybe well-suited to that. But yeah, for myself, I wasn't. So I was like, "I got to come up with something else." And I didn't really want to go back to consulting. So I was like, "Well, maybe I can come up with some kind of software as a service on the side." I was trying to think of what could possibly ... Because I always have ideas, I have tons of things that I want to build. My problem is not a lack of ideas, it was just which would make sense.
John:
And then randomly, I think I saw some flashy thing about Mike Perham, and hitting a million on Sidekiq, or something like that. And I was like, "Huh." And it just gave me a little bit of an idea. I was like, "Well, I wonder if I have an open-source project that could do that." Because I have several. They've been moderately successful. But I was like, "No one's going to pay for HTTP requests." That doesn't really make sense. And I couldn't think of anything else that I had that would really warrant that.
John:
And then I was like, "Oh, Flipper ... " One of the things that you do is, if you use Flipper, you set it up kind of on each project. And it would be really nice. We had a web UI and stuff like that. But that web UI is always per environment, and per project. So if you have eight projects, you got to have it mounted in eight places. And if you have development and then you have staging, and you have production, then the feature data's got to be synced between them. And all these kinds of tedious things. And so I was like, "I wonder if there's a way I could do that."
John:
And so I kind of started looking around, and I saw maybe within a day. I remember writing all the notes. But within a week for sure, I saw LaunchDarkly. And so they're probably I would assume the powerhouse or pretty close, in feature flags right now. And they got eight million in funding, or something like that. And I was like, "Okay, I don't ... " I mean, they've gotten way more since then. But I was like, "Eight million." I was like, "Wow, okay. That means people are actually interested in this as a hosted thing, not just like a ... "
John:
So I kind of looked at their model and how they do it. And they open connections. It's almost like long pulling, but it's not pulling. I think it does actually have push. And I was like, "Well, I don't know if I really want to build that. That sounds really complicated." And I don't think people are going to trust me with an HTTP request in their main request thread. That's not Ruby, for sure. I mean, you have a milliseconds just for being in Ruby and doing net HTTP. So I was like, "I don't know. That seems hard to believe." Obviously you guys have a lot of services like that. Because you have clients that are shipping all kinds of things to you, and you have to come up with ways to do it in the background.
John:
And I'm fairly certain that I've spent a decent amount of time in the Honeybadger gem in the last two years, looking at the thread and stuff. So yeah, I don't know. I was like, "Maybe this would work." And then right after that I saw Split.io came out. And they announced seven million in funding. And again, I didn't want to bootstrap ... Or I didn't want to fund, and I didn't want to go big and that kind of stuff. But I was like, "If other people are paying, or giving large sums of money to people doing this, that seems possible." So I kind of started hacking on some stuff. And at the same time, this guy, Alex Wheeler, he started contributing to Flipper and was like, "It'd be really cool if there was an HTTP API."
John:
And I was like, "That sounds interesting." This was a little bit before the cloud idea. And then he puts a pull request together. And I just was like, "If it was me, I would change these things. But this looks pretty cool." And we had a nice back and forth, and built out a bunch of stuff to just mount a Rack app in Rails or in Sinatra, or whatever. And you just have this HTTP API, and it could do its thing. And I was like, "I bet I could just mount that in a Rails app and then I would have a hosted service." And so it kind of started that way. So it was just for fun. And then one of the other guys that I used to work with at GitHub, John Hoyt. He's John Magic everywhere online. He was like, "I'll built it with you." He was looking just for some fun.
John:
So we started hacking on it. And then the Microsoft stuff went ... And we built it, and it was working. I started using it on Speaker Deck. In the meantime, I had reacquired Speaker Deck back from GitHub. And so I was like ... We were using it there, and we were just kind of pulling every, I don't know, 60 seconds and memoizing things in memory and stuff like that for performance. And it was like, "I think we could do this. There's more to this." And so we were like, "All right. Let's try and make it happen." And so we brought in a couple other people. And then eventually he and some of the other people around the time, and it ended up back as just myself and Steve Smith, who I was partners with at Ordered List, pre GitHub. So yeah, we were like, "All right. Let's try this."
John:
But then it just sat, because Microsoft happened. And that was kind of insane. And so I took a little bit of time off after that. And then rejoined Steve at Box Out. And he had started a new company. So I basically rejoined him at his new company. And then pushing that forward, we kind of just left Flipper in the background. And we started using it a bunch, but didn't do anything with it. And then it was like, "Okay, we have this whole thing, and it's working. We should really do something with it." But we actually launched Speaker Deck before that as a paid thing, in November I think it was when we did it.
John:
And people gave us some money to pay for that pro account. And we were like, "Whoa, this is really fun. We should do this on Flipper. We've had this software that we've been using for two years with no problems. Maybe we should actually slaps some billing on that." So Steve kind of did that over Christmas. And so then we just maybe a month ago released it, so that anybody can sign up-
Josh:
Nice.
John:
... and start using it now.
Josh:
I love that, "Slap some billing on it," by the way.
John:
So that totally comes-
Josh:
That's the way.
John:
Yeah. That comes from Gauges. We actually launched back in the day when we had Harmony, we had CMS. Well, if you have a CMS, you want stats, so we made Gauges.
Josh:
Yeah.
John:
And so what was hilarious was we launched Gauges with a seven day free trial, and no billing. And so we were like, "We got seven days. We can slap a billing system in." So that's probably where ... It's not the first time I've said, "Slap a billing system in."
Josh:
Right on.
Ben:
Yeah, I love that. I've done that. It's like, "Yeah, I'll just push it out there. And if anybody wants to buy it, then I'll figure out how to get some for it." I mean, and if no one wants it, it saved a lot of time, right?
Josh:
Yeah. Who wants to build a billing system that no one's going to-
Ben:
Never gets-
Josh:
... put money into.
Ben:
Yeah.
Josh:
Yeah.
John:
Well, I remember back in the day using Spreedly, which was great. Because they was mostly just like what-
Josh:
Yeah, I remember Spreedly.
John:
Yeah. Webhooks and stuff like that. But then they kind of moved more to being an interface to multiple gateways, rather than the recurring billing stuff. And so I wasn't even sure. And then I went through a seven year entrepreneurial dark period where I worked at a huge company and was focused on that. I didn't pay attention to anything else. But Stripe Billing Portal was awesome. So that actually has covered for Speaker Deck and Flipper, it's made it ... I mean, it's 90% of what we would want. And the other 10% we can live without. So it would just be nice.
Ben:
Yeah. And eventually Stripe will build that 10%.
John:
They will. Yeah, they probably already did, and I just haven't seen it yet. Or it's in the works. It'll be out next month at their rate.
Ben:
Yeah. That's one thing. Over time, I've become much more comfortable with the idea of depending on services for things as opposed to building it myself. And with AWS, it's anything you want, just wait long enough. Eventually it's going to show up. We've been getting more into using DynamoDB, and other services that AWS has. And it's like, "Yeah, I wish had this thing." And then six months later it has that thing now. It's like, "Oh, great."
John:
And that ecosystem is so ... I mean, the only word is, "Rich." I mean, they just let you do everything, SQS, SNS. All those ... Yeah, they're really great.
Ben:
So you got this Flipper Cloud going. And you've got paying customers now because of your billing system, right?
John:
We do, yeah.
Ben:
Yeah.
John:
So we have a few. We launched maybe a week ago ... No, a month in all, I think. Yeah, actually we just had our first person who paid us a second time.
Ben:
Nice.
Josh:
Nice.
John:
This morning. I got ... Thank you, thank you. It was very exciting. I got confused for a second, because I was like, "Oh, a new signup." And then I was like, "Oh, no. This is actually ... It's just the same person signing in again." So I think we have ... I was going to look before. I don't remember. It's probably maybe 300 bucks in seats right now. So I'm really okay with that. Speaker Deck we launched maybe end of November. So basically December, which is kind of Christmas time, and nobody's buying Speaker Deck accounts for Christmas for people, for stocking stuffers.
Ben:
What? Come on.
John:
Yeah, I wasn't sure how that would do. But it's so far maybe 7,000 in revenue that it's generated. And it's up to almost a 1,000 MRR. Which I was like, "If we can get to ... " The expenses on that are around a 1,000, like 1,200 bucks a month or something like that, for the AWS and Heroku, and all those kinds of things. And I was like, "If we can get to that by the end of the year, that would be amazing. Just by the end of the year, if we get to paying the bills on it." It's kind of just a passion project. I just love Speaker Deck, and I can't get rid of it.
John:
And so I'm like, "If it makes money, that's great. If it just breaks even, that'd be fine." And we almost break even three months in. So that was kind of I think what got us really excited about Flipper too. We were like, "Shoot, these things ... There are actually people out there that want to pay for software that solves their problems." It's funny that you can forget that. But Box Out Sports pays the bills and is doing really great. So it was kind of easy for us to kind of forget about that, and just kind of focus on that and not really diversify and try other stuff. So yeah, they're going pretty well. I don't have any complaints. So there's definitely, it's brand new. So people are like, "I don't know if I want to do this or not. It's brand new." Or like, "Tell me more about your pricing."
John:
Because I mean, pricing you guys know is really difficult. That's probably a lot of iterations, yeah, trying to get it right. And so that was a hard one. But I feel like you labor over it and labor over it. And eventually you're like, "The only thing that's going to make a difference is just putting something out there and getting feedback." So that's what we ended up doing. We were like, "You know what? We realize we're putting ourselves out there at half the price of LaunchDarkly and Split, and all these other companies." But we feel like we can make lots of money at this point. So we'll see. We'll just find out. And so far, the feedback has been good. Maybe it's a little bit high for some people. And a little bit, they didn't even notice the price for other people.
Ben:
That's a good place to be. I mean, if no one's complaining about the price, then you're priced too low, I think. Right?
John:
Yep. Yeah.
Josh:
Yeah.
John:
Definitely.
Ben:
And it's nice to be the no-brainer option, I think, for the other end of the spectrum too. So that's a pretty sweet spot to be in.
Josh:
We've found that the pricing project is one that has never ended for us. And I don't think it ever does. I think you always are adjusting your prices to what the current market is, and with competitors and stuff. Yeah.
Ben:
Yeah.
John:
How do you guys do that? Id' be super curious. Every time you roll out, do people kind of ... You roll it out for new people? The new pricing? And everybody else-
Josh:
Yeah.
John:
... just stays on the current plan? Do you have 800 different plans in Stripe, but you only show two of them on the website? I'm just curious.
Josh:
Yeah. That one.
Ben:
All of the above, yeah. Yeah, we try to do right by our customers, right? We don't want to push people into things that ... Like the Comcast thing. Like, "Oh, by the way. Your bill's going up." No one wants that. Right? So yeah, every time we do pricing, we just do it for new people. We typically don't announce it and make a big deal out of it, because we just want to see. Because don't do AB testing on our pricing. So basically, like Josh said, it's basically a full-time thing. Because for months leading up to a change, we're talking about it and thinking about it, and penciling it out. And then we make the change.
Ben:
And then for months after the change, we're like, "Okay, how'd it go? And how's the revenue curve looking?" And then we have had one round of price changes where we did go back to customers who are on our older plans, and asked them to upgrade. When we switched from basically unlimited to metered based on traffic, they were ... One of the reasons we switched to new pricing was because some customers were using a whole lot of traffic and not paying us very much. And so we did go back in some cases and say, "Look, we just can't continue to service your level of traffic. And here are some options. You can fix your bugs, or you could pay us more. Or you could just deal with some of the traffic going away. Well, cut you off at a certain point."
John:
Yeah.
Ben:
And those conversations were great. I think because one, we explained it clearly to people like, "Here's why things are changing." And we reached out, I reached out individually to people. It wasn't this mass thing. It was each account I emailed and said, "Hey, here's your traffic. Here's what we're talking about for new pricing. And here's some ways you can adjust it." And overall, the response was great.
Josh:
The ones that you actually contacted, especially the ones you contacted first, the way you lay it out, showing them their traffic and showing them how much we're making on them. By the end of it, they're apologizing to you.
Ben:
Yeah, yeah.
John:
That's funny.
Josh:
That was a good email.
Ben:
And we've just started ... I think we made our last change around May of last year. And now we're thinking again about, "Is that working? Do we need to make some tweaks." On the receiving end, I don't know if you know about Penboard. It's a bookmarking service.
John:
Yeah, I use it.
Ben:
Yeah. I love Penboard. I signed up way, way back in the day when he was selling it the one-time fee. And I remember his price went up by a penny for every customer that signed up. And I got in, I think it was $2.36 or whatever. And did you get his ... I just got an email from him this past week. And he's-
John:
Oh, interesting.
Ben:
A big, long email. And I was like, "I know what this email is." And yeah, it was like, "Hey, you're a long time supporter. Thank you so much for signing up when I was just charging one-time. But I have ongoing fees, and I need to pay the bills." Et cetera, et cetera. It was a fantastic email. And the pricing, the new pricing is just ridiculously cheap, I'm like, "Sold. Take my money." Right?
John:
Yeah.
Ben:
And so I wrote back to him. I'm like, "Hey, great email. Thank you for explaining this. And happy to pay you more." Yeah, I think we kind of get twisted up in knots over those kinds of things. But if you have customers that love you, and you're treating them fairly, I think people get it. Like, "Oh, yeah. I can see that there's an imbalance here. And yeah, we can make a change."
Ben:
So on pricing, how'd you settle on pricing for Flipper Cloud. What did you consider? Or was it the first thing ... Was it the first thing you thought of we see today? Or did you go through some permutations on that?
John:
Yeah, we went through a lot of permutations. And Steve and I are the only two in the company. So he does most of the design and stuff like that. I do most of the backend, and we both cross over. We're real big on standards and stuff. So using the same things in all of our apps, so we can work and stuff like that. But business stuff we always do together. Which is good, because we have, like you guys obviously, probably find that you have different things that you bring to the table.
John:
So I think the goals is, we just want everybody who's using Flipper to use this. Mostly just because it will make your Flipper life better. If you're already using Flipper, it will make your life better. So we want people to use it. We don't want them to not want to use it just because it's paid versus open-source, or something like that. So that's one thing we've factored in.
John:
And so I think initially, we were like, "Well, what if we just came up with a small plan of some sort. Three plans," like everybody does. Like, whatever, nickel, bronze and gold. Or that kind of idea. On Box Out, we have Basic, Plus and Premium. So I mean again, it's just you make three plans, you bucket people in and move on with life. So we started with that. And the thing we were noticing was, LaunchDarkly, Split, Rollout.io which is not Rollout that we were talking about earlier. But there's several of these that exist and have pricing. And I went down through all of them. I'm sure you also do this. But did a little analysis. And I'm like, "These people charge a lot of money for feature flags." I was like, "It's crazy."
John:
I mean, it's not crazy. The people who are paying it is totally sane. But I was like ... It's just kind of mind blowing to me. I'm like, "Okay, I ... " I pay whatever, 10, 15 bucks a month, or maybe 25 bucks a month for some of the organizations that I have on GitHub. And I'm like, "If I charge $20 or $40, or $50 for one seat in a feature flag app, are they getting more value from that, or from GitHub?" It's difficult for me to say something that's been around for eight years, acquired by Microsoft and all this stuff that it's almost like how can I charge that? But then I try to remind myself like, "Well, yeah. But they have millions and millions of customers."
John:
So that was kind of a lot of the analysis was just like, "Well, what price point are people at?" Because you don't want to price super cheap, and then have people be like, "Well, they must be cheap." And you don't want to price super expensive if you don't feel you're delivering that kind of value to people. So I looked at what the market was at, which seems to be somewhere in the 40 to 50 a seat range, with different levels of freemium to get people in. And then after that, Steve and I talk and we were like, "We just want people to use this. So what if we just have a 25 bucks a month plan for some number of two people?"
John:
And then the value with cloud, I feel like is ... That's the hard thing is it's brand new. So how do you tell what the value is? But for you guys, you can have metered billing, which that makes sense. Because the more I use of your service, the more value I'm getting. And so the more money you should make. And obviously there's other things you probably differentiate on. But for us, we're ... Right now initially what we're offering is everything kind of in one place. And we help keep it all in sync. And some of that kind of stuff. And it really doesn't cost us a lot to do that. We have ideas down the road for analytics and other things that will cost a lot. So we can figure that out later.
John:
But we didn't really have a good value metric other than the number of developers. So the number of people that are using it, if you have two developers, it is less useful ... Or one developer, it's less useful to have it than if you have 10, 20, whatever. So we were like, "Well, if that's going to be the case, having a $25 plan, and then ... " I don't know, whatever. A $500 plan, or maybe a 250, $300 plan and a five or $600 plan, and then a, "Call us for more," kind of a thing. Where going from 25 to 300 or something just felt like a lot. And we didn't want to have eight different plans in between, and come up with Adamantium, and every other metal.
John:
So it was like, "Okay. Well, what are we going to do?" Realistically, we've always hated per seat pricing. Steve and I both have just kind of ... I remember when GitHub switched over to per seat pricing. And I was like, "I mean, whatever's in the interest of a shareholder. Cool." But I really hate per seat pricing. But I was like, "You know what? It just really actually makes the most sense." It is the thing that gives your organization more value. In this case, more developers using it means more things to keep in sync. And more things to keep track of changes, and stuff for, and more things where permissions matter.
John:
So I was like, "Okay, that actually makes the most sense. We need to do that. What do we charge?" And so we went back and forth on that for a while. Do we just make it 10 bucks a month? And hopefully everybody's just like, "Well, that's a no-brainer. I'll just do it." Or do we make it 50 bucks a month like everybody else? But we were like, "That just felt like too much." So we were like, "Okay, let's just pick kind of in the middle." So it was really scientific.
Josh:
Yeah.
Ben:
Yeah.
John:
A whole bunch of research. Yeah. See what other people are actually charging for it, what kind of value people you assume they're getting. And find the value metric that makes the most sense. Interestingly, a lot of people for feature flags charge based on a monthly active users in your app. Because they assume you're going to pass that user through every feature flag. And the reason they do that I think is just because of again, the analytics side. If you're actually sending each feature check, that's a lot of data. Because you can have ... I mean, GitHub would have hundreds of feature checks on any page rendering. And some of them were in a loop where you're checking it for every single listed org or issue, or things like that.
John:
So it's easy to generate a lot of data like maybe an exception app would get. And if that was the case, then yeah, you want to bill based on that. But we're like, "We're not doing anything with that right now. So we're just going to start with what makes sense." It's like, "Here's the cheapest we think it would make sense at. Here's where everybody else is at. And we're just going to kind of go in the middle. And we'll see how adoption goes." And then if it goes well, cool. If it's struggling, then ... We've got a few people who are like, "I don't know if I can quite pay that much." "Okay. Here's your discount." Like, "I just want you using this. Give me feedback. That's fine." And then we'll hammer out the pricing, because we basically just threw a dart after hours of discussion.
Ben:
Right.
Josh:
Yeah.
Ben:
Yeah, we've been there. We've done that. And yeah, I too dislike the per seat pricing. But I agree, it makes a lot of sense in this case. And when you get to that point when you do the analytics stuff, you could ... So as an add-on. It's like, "Oh, you want the data pack too? There you go. All right?" When we started Honeybadger, we were really, I think, turned off by what Airbrake had done with their pricing, and being really limited based on traffic. And so that's why we decided like, "We don't want to do that." And we decided the key limiting factor that we could tier on was number of projects.
Ben:
We figured bigger teams would have more projects. And so they can pay more. And that worked out, except again, for those people that only had one project and they were sending us all this traffic. What really changed our minds even more than that, because we were just willing to bear that. But what really came down to it was microservices became a thing. And these small teams would all of a sudden have five and 10, and 20 little microservice apps they want to track independently. And our pricing just didn't work for them. Like, "I can't spend that much just for my little two person team," right? And so I was like, "Yeah, that does make sense." And so we flipped that around to that, "Okay. Well, if your overall traffic is low, then you don't have to pay us so much."
John:
What's funny is I looked at your website for a while last night, and I looked at everything but the pricing. And A, first off, I love it. There's so much good stuff on it. Super fantastic work. I just enjoyed reading it. I'm listening to some previous episodes and looking ... I felt like I completely immersed, I'm full on Honeybadger this morning.
Josh:
Awesome.
John:
But yeah. So you are meter based on basically what the traffic that people send to you. That's the ... Okay. I just wanted to confirm that. Cool.
Ben:
Yeah. Yeah, and it's tough. Because if you're a new Honeybadger customer and you're coming, you're like, "What pricing makes sense for me?" You don't know how much traffic you're sending. And it's hard. With cell phone management, you're like, "Oh, I know how much I talk per month," right? Or with user seat pricing you're like, "Well, I know how many people I have on my team." Right? It's easy.
John:
Yeah.
Ben:
But with the traffic it's like, "I don't know." And so that's another reason why we didn't launch with that.
Josh:
And this was back, whatever, nine years ago when very few people were doing metered billing too. So it was a new concept to everyone.
Ben:
Yeah.
Josh:
So yeah.
John:
I remember talking about that a lot with Gauges, metered billing. We ended up doing three plans and just saying, "Contact us for more." And then we had some hidden plans in Spreedly that didn't show up on our website, but for bigger customers and stuff like that.
Josh:
Yeah.
John:
But yeah, metered I feel like works better with some kind of in that instance. Look, it's cool. If you just ... Somebody does this. They do a two month rolling average, or a three month rolling average, which I thought was cool too. Like, "If you use a little more one month, but less the next months, we don't really care." I feel like that's kind of been our approach to billing is like, "Look, we got to charge you something so that it's worth it for us to keep doing it. But we want this to make sense for both of us. We're not going to ... " Not to pun. But we're not going to badger you into higher prices, or whatever.
Josh:
Got to write that down.
John:
Exactly. So yeah, I feel like that's kind of how I always have approached it. And so metered pricing is interesting with some kind of forgiveness built into it. I feel I like that idea.
Josh:
That forgiveness was what was missing with Airbrake I think primarily. Because back then, it was ... If you did metered billing, it was like, "This is the number with whatever events that you get." And then it's like, "I just cut you off," basically. I just remember having those issues where if you hit the limit, then it was like, "You're on your own." And that feels terrible from a user, customer standpoint.
Ben:
Yeah.
John:
Yeah. You could hit that with a really bad exception in high traffic.
Josh:
Yeah.
John:
I mean, I remember that with everything in Haystack, it was rolled up based on just a super simple algorithm. And if you don't have that, you can have one exception that just blows your entire limit. We had that happen at Box Out. I don't even remember what the exception was. But it literally just ate through our entire Century thing. And I was like, "Oh."
Josh:
Yeah.
John:
So then we upgraded the plan so we could get more. I think it was JavaScript tracking. We didn't even care about it. So we just-.
Josh:
It's always JavaScript.
Josh:
It's always JavaScript.
John:
Yeah. I mean-
Josh:
It's always JavaScript.
John:
Yes. yes.
Josh:
It was probably some old legacy version of IE that ... Yeah.
John:
Yep.
Ben:
Yeah. The forgiveness is key, I think, to having a great relationship with your customer. And the truth is, we're softies. We have people on a regular basis emailing us and saying, "Hey, I'm sorry I just sent you a million and a half errors yesterday. And my plan limit is a 150,000. But can I just get by for the rest of the month without paying you more?" And it's like, "All right. Fine." And so we flip a little switch that allows them to have the rest of the traffic for the month.
John:
What's that switch? Is it just you just wrote some code, and you cached the code? Or it's fast enough so it's fine?
Ben:
Yeah. So our internal admin tool just pushes a ... I guess it's a Redis key.
John:
Okay.
Ben:
So we have a daily check. So then we look back at the usage, and say, "Okay. Well, are you past your quota yet?" And if you're past your quota, then we'll add a limit. And so we just have a flag in Redis that says, "Oh, this customer's exempt until such and such time." And so that data check will be like, "Just ignore them."
Josh:
Yeah.
John:
Yeah, it works.
Josh:
We abuse Redis a lot like that.
Ben:
We do. Quite a lot.
John:
I mean, everybody does. And pardon me, I feel like that's a huge ... That's where Redis gets a lot of its market share is that kind of easy quick stuff. So I've been thinking about that a lot. And we don't have to talk about it. But in Flipper, I think I've been calling it controls. So there are there's ... So we have it in Speaker Deck. In Speaker Deck we have Quality. And Quality defaults to 90. But some people are like, "These aren't clear enough." And so we're like, "Okay, 100." And before, we couldn't do that. And I was like, "Well, I need some way to just say this is the setting for this person. If it doesn't exist for that, then fall back to a default."
John:
And I was like, "That would actually fit ... " Flipper has, "Flipper.enabled?," Feature name. And then Actor. That's the same as a config. It's just instead of being a true/false response, you're returning some other typed piece of data. I feel like that would be ... Every app that I've worked on, I wanted that. And Flipper already has ... It's basically just like a key value thing with some Ruby logic on top of it. So now that we have syncing and other stuff, I'm like, "I totally just need to build that in, because I desperately want it."
John:
On Box Out, I want to say like, "What's your graphic limit? Or what kind of ... " It's not for secrets. It's for configuration. Things that you want to hot reload. You don't want to ... And then you just memoization and caching, and all those layers built in, just like they're built into Flipper or whatever you use. So yeah.
Josh:
Yeah.
John:
That was why I asked about the switch, because-
Josh:
That's cool.
John:
... I love that idea of controls, and I'm always curious how people do it. And I feel most people do it in Redis. And they do it because it's easy there. But not necessarily that they prefer to have that be the place. Because then you got to remember to clean it up if you ... Otherwise that will just keep growing. But yeah, sorry.
Ben:
No, that's cool. I love where you're going with that. Because in every billing system I've ever written, there's always this notion of limits per plan. A plan has certain project limit or a user limit, whatever. But you always want to be able to override that and comp something to a customer, right? "Oh, I just need another user." "Okay. Well, you can have another one."
John:
Yeah.
Ben:
And so there's always the user limit field on the plans table, let's say. But then there's also a user limit field on the accounts table, so that you can override. So if that's nil, it just falls back to the plan. But if they've got a value, then use that value instead. So you could totally do something like that with Flipper, kind of thing I think.
John:
Yeah. I definitely, I have it in my mind. I just have to sit down and do it. The only thing that's hard with that is, you have an explosion in your cardinality, because if you can have one setting for all one million users. All the Flipper data is designed to be able to be, basically you can read all your features and all their data in one request. And then the dynamic parts of it go in Ruby. Or the dynamic parts some day will go in whatever other language support we add.
John:
And so I try to control cardinality just because I've worked on GitHub, and Words With Friends before that. So you work on high traffic things, and you see how things fail. It's, "Well, you don't have a limit." And so you run into that with Amazon services, or anything where you'll have a limit. But you can email them, and then they'll up the limit. And so it's like they try to keep people from hurting themselves. But if they really know what they're doing, they'll know, "Yeah, they'll just turn it up."
John:
And so I just feel like it's a core part of every app, and there's no library or anything that kind of does it just from scratch in a way to make sure that you're not doing N+1 lookups all over the place for every configuration value, and stuff like that. So I desperately want it, so I feel like I'm going to build it in probably sooner rather than later. And it's a differentiator. There's only one or two other places that are doing that. I think ConfigCat is one that I saw that seems to do something kind of like that. But yeah, so that's why I was curious.
Ben:
Yeah, cardinality is a killer. Way back in the day when we first attempted to use Elasticsearch to index our customers' exception data, every exception is a unique snowflake, right? And so they all have just ... The union set is very small across all customers. And so Elasticsearch has of course limits on how many items it's going to track. And once you hit that limit, tough, sorry. Because they have to distribute that all around the cluster. So they have to limit that. And so we had to come up with some clever ways to normalize that data, and feed it into Elasticsearch in a way that won't blow up that cardinality.
John:
That's cool. It's tricky.
Ben:
Yeah. I should blog about that some day.
John:
You should. I would read it.
Ben:
So one more thing on price. So on the ... You got the Flipper open-source thing. And you've got the Flipper Cloud, and you've got the self-hosted option with the cloud on your pricing page. Did you think about other ways ... Could you talk about Mike? Mike was an inspiration. And Mike has a particular licensing scheme, right? There's open-source. And it's basically an open-core model, right? You want more features or you want a different license, then you pay for that, right? So did you think about those kind of alternatives for Flipper? And what made you decide on the hosted thing was right for you?
John:
Yeah. I definitely did. That's a really good question. I feel like ... I saw Mike's thing, so that was my first thought. And also, Robert, the GraphQL gem, he has the same kind of, I think as Mike. Where you can pay a little bit and get a different license or get more features, resolvers, cachers, all those things. I was trying to think through that with Flipper. And I'm like, "Well, I could definitely do that. But what a thing that's painful with open-source Flipper?" And that's the problem I would want to solve. And I mean, the number one things that people kind of brought up were audit logging, or some kind of audit history. They just want to know who did what, when.
John:
And then the other pain, a big one is everything is per project and per environment. And so everybody's got rake tasks, and sync scripts, and all these kind of stuff. So it's like ... I mean, especially when it gets like GitHub where you have hundreds of developers. And they're all creating features, and going to town. And you go on your local laptop to do something, and you're like, "I don't even know which feature I need to turn on to make this work, to see the feature." And so then you got to go through the code and figure out where it's hidden, and figure out what features get used, and how they need to be enabled, and all this kind of stuff.
John:
And I feel like those were the two things that we saw as the biggest pain points were moving feature data through your environments, and having a history of what's going on. Those were the two key things. And then the other thing that I feel like long term is huge, is the analytics on the information side that can be done. I've said it before, but I'm the kind of person that I played NBA Tecmo basketball, and Tecmo football on Nintendo growing up. And Tecmo football didn't track any stats, so I paused after every play and I kept track of passing attempts, rushing attempts, yards, tackles.
Ben:
It's hard work.
John:
NBA Basketball ... Yeah, it was better, because they kept track of everything. But they also kept track of rebounds, but they didn't do them correctly. I started noticing problem ... I'm like, "I know I had more than 10 rebounds in this game. This isn't right." And so I started keeping ... And I had my own rebound ledger. So I've always been into stats, I guess. And again-
Ben:
Apparently.
John:
Yeah. Thinking through what the problems were with open-source is, well A, you want to know, is a feature being checked or not? And if you set it to 25% of actors, kind of how many people does that mean? And is it actually returning that? And at GitHub, we used StatsD and stuff like that to kind of figure some of that out. And then we also eventually used other things, like Datadog and stuff like that. But I don't necessarily have that on every project that I set up, and stuff.
John:
But these are things that I want every time I use Flipper. And so if I'm going to want them every time. And probably other people are struggling with them every time, then those probably ... And those are the things that are the hardest to build, because they're unique to the setup. So if you want to have analytics tracking, you got to have some thread in the background that's keeping track of stats and shipping off events, and doing that kind of stuff to some kind of end point. And then that end point is going to have to do a lot of work to store the raw data, crunch up reports.
John:
Again, you guys, totally familiar with that. But that's not something you just want to solve in open-source. I was just that ... Or for each customer. I don't want to write an analytics roll up adapter to work with Redis and to work with Mongo, and to work with Postgress. There's some things that just make sense to centralize. And so I think that's why I went that route. And also, that was what I had done in the past with Harmony and Gauges was SaaS. I just understood that ... I've never licensed anything. I don't even look at licenses. I mean, other than legality just to make sure I'm not breaking laws. But yeah. I mean, I use MIT, because everybody uses MIT in Ruby. There's probably other licenses, I don't know what they are.
Josh:
Yeah.
John:
So I think that's kind of why I didn't go down that route.
Josh:
Have you thought about AB testing features?
John:
Yeah. That comes up. I feel like ... I mean, for me where I sit on that is, I don't AB test stuff. Like you guys said, you don't do it either. And I feel like I don't really have the desire. And I kind of alluded to this earlier, but just to sit there and micromanage the revenue or the color of a button, or all these kinds of ... I just don't care about that. So it doesn't interest me. I don't personally do it. I wouldn't know how to solve it right for people. There's other services out there, Optimizely, et cetera that are huge.
John:
But everybody that uses feature flags immediately is like, "Oh, this is the same thing as AB tests. Why doesn't this do AB tests?" Or I'm already doing AB tests, et cetera. So I've already ... I mean, there's been several people that have asked. And so probably some day. If your customers want something, you build it. I wouldn't know how to build it correctly for them right now. And that would detract from just trying to get people who are using Flipper currently onboard, which is goal number one is just if you're using Flipper, just make it a no-brainer that this is something you use.
John:
So that's the only reason why I haven't is I'm not sure how much I believe in it. And if people want to do it, that's totally fine. And I could see it being supported at some point. But it just feels, yeah, different. To me, they're different things. And maybe what you do is there's a better way with once you have those raw stats, you use some sort of export Zapier, or whatever it is. You have glue between two disparate systems that they use the data. But I don't know that I want to have an AB testing company. But I actually, a year after I left GitHub, the people who were working on maintaining Flipper internally were like, "Hey, we're looking at doing a lot of AB testing." And they had two different things, which-
Josh:
Didn't GitHub release a tool that did AB testing in Ruby or something? I feel like there was something back in the day?
John:
Scientist?
Ben:
Scientist.
Josh:
Yeah, Scientist. Yeah. I was going to say five years ago, or within the past five years I saw something.
John:
It's more for code changes to make sure that you can try-
Josh:
Okay. Yeah.
John:
... the old way and the new way, and test the performance-
Josh:
That's right. Yeah.
John:
... before they go out. Yep. But it's the same. That I love. And I feel like that's totally up Flipper's alley too is this idea-
Josh:
Okay. Yeah.
John:
... of server side tools to help you release software more safely. To me, I love that. I'm just less excited about the AB side.
Josh:
Gotcha. Okay. Yeah, the Scientist thing was maybe a little bit closer to what I was thinking, versus product-
John:
Oh, gotcha.
Josh:
... feature maybe. I don't know. Yeah, I probably could phrased it a little better. It's just you said-
John:
I didn't know about that at all.
Josh:
... you love stats. And I thought, "How can I get this guy collecting more stats?"
John:
What crazy is I've never thought of the Scientist functionality in Flipper. But that makes total sense.
Josh:
It seems similar-
John:
Yeah.
Josh:
... to me.
John:
I think I owe you a percentage of the company now.
Josh:
Totally. The first one's free.
Ben:
Yeah. I got to put a plug in for Scientist. We recently used that, because we were upgrading our elastic search cluster from six to seven. And so we spun up a new cluster, filled it with data, and then we were like, "Okay, let's see. How does it work compared to our old cluster?"
Josh:
I didn't realize we used Scientist for that. That's cool.
Ben:
Yeah. It's great.
Josh:
Nice.
Ben:
So it really helped me realize, "Oh, okay. I won't blow up the world if I deploy this new thing."
Josh:
Yeah.
Ben:
It's quite nice.
John:
Yeah. And the key thing with that is that I love having the AB execution, and deciding whether or not to run the B. And that was even ... I see that as it's more like A to B down, rather than A or B. We're always going to return A, but sometimes we'll do this other one. And I mean, I've been using that so much. I mean, it's really valuable. And the biggest thing I miss from GitHub honestly, was back when we still had StatsD and Graphite, and metrics whatever. Some magic that all the smarter people made internally just for us, or whatever.
John:
I miss that. I mean, you could just throw anything in StatsD that you wanted, and you never had to think about ... It was great. Now that I'm outside, I'm like, "I just want to click a button and just have that be on a thing." But not have to worry about, "Do I have a 100 metrics, or 200 metrics? Or 300?" I just want to ... If it fills up the disk, I'll just delete the disk and start over. It's fine. I just want to click a button and have it appear. And so I can just send Scientist stats and all that.
Ben:
We love that. So we use StatsD and send it out to Librato.
John:
Oh, nice.
Ben:
It's wonderful. I don't have to care. I just pay whatever per month, and they keep track of all of the stuff. And I can make nice little charts whenever I want on demand. And it's great.
John:
I need to check that out in more detail.
Ben:
I think they've rolled that out now. I don't think they actually sell Librato separately.
John:
Oh, okay.
Ben:
I don't know. Have to check that out. But yeah, it's cool stuff. Well, yeah. If Josh just gave you a huge business opportunity there, then yeah, we'll wait for that check in the mail.
John:
Yep. Just don't hold your breath.
Ben:
Well, John, thanks so much for joining us. This has been awesome. This has been a lot of fun.
Josh:
Yeah. A lot of fun.
John:
Yeah. This was great.
Ben:
So for everyone who wants to keep track of you, what's the best way? Twitter, your website? What do you like?
John:
Yeah, either one of those. I mean, anything I put on my website I'll eventually put on Twitter too. So either one of them is fine. And I'm now a proud Ghost user, paying Ghost user.
Josh:
Nice.
Ben:
Cool.
Josh:
What's your Twitter handle, by the way?
John:
It's @jnunemaker. Which most likely you can't spell. But that's what the show notes are for.
Ben:
That's right.
Josh:
Yeah. We'll put it in the show notes.
Ben:
Well, thanks, John. And best of luck to you with Flipper Cloud. I'm going to go check it out again. And maybe we'll switch from Rollout today.
John:
Yeah. Hey, if you need somebody to pair with, you know where to go.
Ben:
All right. It sounds good.
Josh:
Nice.
Ben:
Thanks. Well, this has been an episode of FounderQuest. If you want to write for the blog, I'll have to plug that for Star. You know where to go, because you've heard the podcast's outro so many times by now. If you want to check out John, we'll have some links in the show notes to his Twitter and his blog, and Flipper Cloud. And if you want to do some work for Honeybadger, Josh has been hiring like crazy.
Josh:
Yeah.
Ben:
And got a whole bunch-
Josh:
Still could use some more.
Ben:
... of contractors. Kevin said in Slack the other day, "Yeah, they're like Pokemon. Got to collect them all."
Josh:
Collect them all.
Ben:
But thanks so much for listening. And hope you all have a great day.