On Sunday, we had a service outage. Jon and Justin talk about what happened:
TLS / SSL day of horror
Sunday morning Justin started getting messages from a few Android users that they couldn’t download episodes.
Then, the afternoon hits, and we started getting a flood of tweets, DMs, text messages, and support requests.
We talk about how we fixed it, and what we'd like to change for next time.
Also: how we're earning 29% of our SaaS revenue. (A channel we didn't expect)
★ Thanks to our sponsors:
Clubhouse.io: Clubhouse is the first project management platform for software development that brings everyone together. It's designed for developers, but product folks, marketing, support folks love using it. Get two months free: clubhouse.io/build.
Balsamiq.cloud: They make a low-fidelity wireframing tool, specifically geared for non-designers. Get in the zone, and I feel creative right away. Try out their free trial: balsamiq.cloud.
On Sunday, we had a service outage. Jon and Justin talk about what happened:
TLS / SSL day of horror
Sunday morning Justin started getting messages from a few Android users that they couldn’t download episodes.
Then, the afternoon hits, and we started getting a flood of tweets, DMs, text messages, and support requests.
We talk about how we fixed it, and what we'd like to change for next time.
Also: how we're earning 29% of our SaaS revenue. (A channel we didn't expect)
★ Thanks to our sponsors:
Clubhouse.io: Clubhouse is the first project management platform for software development that brings everyone together. It's designed for developers, but product folks, marketing, support folks love using it. Get two months free: clubhouse.io/build.
Balsamiq.cloud: They make a low-fidelity wireframing tool, specifically geared for non-designers. Get in the zone, and I feel creative right away. Try out their free trial: balsamiq.cloud.
Interested in building your own SaaS company? Follow the journey of Transistor.fm as they bootstrap a podcast hosting startup.
Justin:
This week's episode is brought to you by Clubhouse. Clubhouse is project management software. That's a delight to use. It's fast. It's easy.
Justin:
It allows you to zoom in on your tasks and focus. It has tons of integrations, GitHub, Slack, Sentry. And, lots of people are switching to it. I I posted about it on LinkedIn and got tons of folks like Rodolfo rem Ramirez. He's a COO at Swivel and Nathan Palmer, founder of products.
Justin:
They responded to say they've switched to Clubhouse as well. Find out what all the buzz is about. Go to clubhouse. Io/build and get 2 months
Jon:
free. Hey, everyone. Welcome to build your SaaS. This is the behind the scenes story of building a web app in 2019. I'm John Buda, a software engineer.
Justin:
And I'm Justin Jackson. I do product and marketing. Follow along as we build transistor dotfm.
Jon:
And probably making me few mistakes and fix stuff on a Sunday.
Justin:
We had a hard weekend, folks. We're
Jon:
tired.
Justin:
Yeah. We're we're gonna get into it. I think that the working title for this episode is worst day ever. Maybe I'll just say one thing, which is Yeah. This is how it happened.
Justin:
This is how it started. Is, Sunday morning. So we're recording this on Monday. So this is yesterday. It feels like this happened a week ago.
Justin:
Yesterday morning, I started getting messages from Android users saying they couldn't download episodes. And I'm trying to figure it out. It just seems like it's happening. It's only happening on Android. And at first, I thought it was only certain apps.
Justin:
We had recently dealt with this other thing with PodTrak where TLS 1.3, which is like the new SSL. Right?
Jon:
Yeah. It's yeah. It's the new, the new protocol that's, like, better and faster than the previous one and not in our, the web server we use, like, automatically kinda upgraded to this and made it the default.
Justin:
Okay. Yeah. So we so you had upgraded some stuff. Now we get the new TLS 1.3, which is cutting edge, kind of security.
Jon:
Although it's not even that new. Like, it's Yeah. It's just that stuff is so slow to adopt.
Justin:
Yeah. I think it was released officially August 2018. Yeah. So initially, I'm thinking it's just TLS incompatibility. Lots of Right.
Justin:
Lot like Android hasn't upgraded their platform to TLS 1.3. So I thought it was that. So I'm, like, answering, you know, maybe 5 to 10 issues from people around what I think is this TMS issue. And then in the afternoon, I just start getting like a ton of messages. And, you know, depending on the weekend, like some weekends, I'm away from my computer skiing.
Jon:
Yep.
Justin:
But today I was, you know, yes. Sorry. Yesterday, I was just I was on my computer. And so I'm getting all these messages. And I'm like, wait a second.
Justin:
There's something's going on. And, people are saying you're, it looks like your SSL cert is expired. And so I'm cert message, trying to message John and John is like doing some sort of meditation retreat in the woods or I cannot get I cannot get a hold of John. And, I was freaking out a little bit, but not not too bad. And so
Jon:
Justin thought I sabotaged the system and ran away.
Justin:
Yeah. Ran away. So why don't you take it from here? So what, yeah, what happened and
Jon:
So what happened yeah. It was basically like a chain reaction of bad events that happened at the same time, I think.
Justin:
Yeah.
Jon:
The the thing related to TLS 1.3 is is a problem Mhmm. For in certain situations, though our our server does also support TLS 1.2, like, simultaneously. It's sort of like
Justin:
Okay. It'll roll back to that?
Jon:
When someone when someone when another site or a browser requests something from our site, it will try to negotiate, like, the best the best protocol that they both that you both support Mhmm. And use that. But it's still an issue in certain instances. Yeah. So I was like, okay.
Jon:
Well, that that can be fixed. Whatever. And then so we we upgraded a a while ago back in December to, Caddy, which is a web server that does automatic, SSL certificates through Let's Encrypt. Yep. So we can do custom domains for our customers on their websites.
Jon:
But that certificate that the Caddy also provides our SSL certificate for transistor.fm. Mhmm. And so the one benefit of Caddy is that it's supposed to automatically renew your Let's Encrypt certificates, like, 30 days before they expire because they last for 90 days. Mhmm. Which was you know, I think we I rolled this out around Christmas, which would be 90 days, you know Mhmm.
Jon:
Previous Yeah. To today, this weekend. Yeah.
Justin:
Does that
Jon:
make sense?
Justin:
The timing lines up.
Jon:
For some reason, which I'm still determining, Caddy did not automatically renew our certificate. Yeah. And I had gotten emails from Let's Encrypt that was like, hey. Your certificate's expiring in 30 days. And I'm like, yeah.
Jon:
That's fine.
Justin:
Mhmm.
Jon:
It'll be handled.
Justin:
Yeah. We got it handled.
Jon:
And it didn't. Sure didn't.
Justin:
So By the way, I think this is this is just a a rite of passage for any web app.
Jon:
But it just it's just a it's just frustrating. Like, it just sucks that it was down for as long as it was because even even fixing it should have been simple, but but that took hours like, a couple hours longer than I wanted to for a number of reasons. Like, I just couldn't there was no easy way to say, hey. Regenerate this Mhmm. Right now.
Justin:
Yeah. Yeah.
Jon:
So I kinda had to, like, wipe things clean as far as, like, not any of our data or any like, everyone's podcasts are obviously fine, but sort of had to wipe, like, all the certificates that were stored on our servers temporarily, and just let them kinda regenerate as they were being requested. Gotcha. Which is a fine thing to do. Mhmm. In the process of that, I also finally purchased a commercial license for Caddy, which we probably should have done since we were using it commercially, which which does include, some basic support.
Jon:
So I've been emailing back and forth them trying to, like, you know, I you know, vlogging every we're logging everything, like, going through our logs and being like, hey. You know, what's this mean? Yeah. And, what does this what does this error mean, and why doesn't this SSL handshake work? And, basically trying to get things back so that, you know, most, if not all, secure requests work properly.
Justin:
Yeah.
Jon:
So, I mean, I was, like, already tired yesterday. Yeah. And then to have to do that on a Sunday night is it's just it's frustrating. I I, you know, I feel bad for our customers who, you know Mhmm. Had, like, couple couple hours of downtime.
Jon:
And even today, I think couple of things aren't working properly. But mostly, it's back to normal.
Justin:
Yeah. So yeah.
Jon:
Just like you you're doing you're trying to fix the stuff. Like, you're trying to, like, do as fast as you can, and you're and you're tired. You're like, I don't wanna make any mistakes. Mhmm. It's just yeah.
Jon:
Yeah. It's rough.
Justin:
One of the nice things about sharing our journey is that John and I aren't trying to pretend to be like a big corporate machine. Like, we're real human beings that are trying to build the best thing we can for people. And we really do like, the first thing John said after this was fixed is I feel so bad for our customers. Like, that is just guilting himself. Like, I feel bad that this happened.
Justin:
That's that amount of downtime is just not acceptable.
Jon:
Yeah. I just I I wouldn't I wouldn't want that for, like, a platform that I'm paying for.
Justin:
Yeah.
Jon:
Especially if I'm, you know, running a business through it or whatever
Justin:
Yeah.
Jon:
As as some people are.
Justin:
Now on the flip side, our customers were incredibly gracious.
Jon:
Yeah.
Justin:
And I think the advantage we had that day is that I was around my computer, And, I was just able to answer a lot of people in real time and kind of do, I was able to handle all the incoming Triage? Triage. Yeah. I was just on triage. And, you know, that worked.
Justin:
And I I my whole thing is I just tried to be as honest as I could and as as as fast as I could with responses. So
Jon:
Yeah. And I I am definitely grateful for that.
Justin:
And I think that's a great that people were understanding. People, you know, they they reached out because they cared about us. Like, they were saying, hey, just so you know. And I couldn't believe, like, it was actually the nice thing is I I got a a lot of messages from listeners. So, like, my buddy Sean McCabe texted me.
Justin:
Hey, John Justin. I'm trying to download an episode from Transistor.
Jon:
Uh-oh.
Justin:
And I'm getting all these things. I'm like, oh, we already know about it. Thanks so much for letting me know. Because it shows that people care. It wasn't like he was like, no no one reached out in a mean and cynical way.
Justin:
They're all saying, hey, Just so you know, it looks like, you know, your certificate expired.
Jon:
Yeah.
Justin:
And, that was that was, in some ways, just really nice.
Jon:
Yeah. It's not like we have, you know, a full DevOps team over here working around the clock Mhmm. From different time zones.
Justin:
Yeah. Yes. Exactly. And and Maybe someday. But, you know And to be fair, if from even most apps that you need to be on all the time, for it to go down on a Sunday is not the worst thing in the world.
Jon:
Yeah.
Justin:
All the podcast apps just basic this isn't exactly how it works, but they just wait until the app becomes the the feed becomes available again. So the fact that feeds were unreadable for a couple hours doesn't mean no one didn't get their podcast downloaded as soon as it was back up. Right. Everyone
Jon:
And it doesn't it doesn't mean that these other apps or platforms will, like, remove your feed from circulation.
Justin:
No. No. No. I checked Apple Podcasts right during it, and they were showing errors. And then afterwards, they, you know, they, crawl the feed again.
Justin:
They
Jon:
Yeah.
Justin:
They, you know, fetch the feed again. And once they see that it's valid, they're good to go. So
Jon:
Yeah. So all in all, yeah. Definitely, some lessons learned. It was it was definitely stressful. I think we'll certainly be in a better place.
Jon:
Like, I I there was just nothing to tell me that renewals were not being automatically done. Yeah. Yeah. Which is kind of annoying.
Justin:
Yes. Yeah. Yeah. It's the the yeah. I mean, we're we're it's hard to figure out what could have because I I run Let's Encrypt on a bunch of my stuff.
Justin:
So in one case, I've just had someone write a script that just runs on DigitalOcean that just renews it for me.
Jon:
Mhmm.
Justin:
In the other case, well, Flywheel just has something on their end that runs renewal scripts, and WP Engine has scripts on their end. But in all three of those cases, I'm depending on an external service to automatically renew those certificates.
Jon:
Or hopefully, tell you if it it didn't renew.
Justin:
Yes. Yeah. Yeah. Hopefully tell you if it didn't renew. So that there's probably something and I'm sure we're gonna get folks and feel free folks to if you have advice or tips.
Justin:
You know, some folks were saying they only use certs from Cloudflare because they're 15 years. And, you know, there might be reasons why we can or cannot do that. But, I I still love Let's Encrypt. I think that their mission is is, I support them in their mission to make SSL free for folks.
Jon:
Yeah. I think it's it's great. It's a lot you know, it's really, obviously, we couldn't really do free SSL for our custom sites without it.
Jason:
Mhmm.
Jon:
That said, it it is definitely still new. Mhmm. So is so is Caddy, the web server we're using. Yeah. Though I haven't you know, I really have no complaints about it aside from, you know, not renewing our certificates.
Jon:
But, but they've been, you know, I they've been good. They've been really Yeah. Helpful and quick to respond over email. So
Justin:
Yeah. Yeah. There's there's also I think there has been, there's a tension here, like, just like so many things. So on one hand, it is like, I love it when David Hanna Meyer Hansen, like, when it when they have downtime at base camp, he gets on Twitter and blogs and everything. And he says, I am so sorry this happened.
Justin:
We do not want this to happen again. This is why it happened. And this is, you know, what we're doing to remedy all that stuff. And I want us to be the kind of company that does that. We are really sorry that this happened for our customers.
Justin:
On the other hand, there's this tension, which is this is let's not, over dramatize things either that, you know, this is the Internet. The Internet net is tied together by duct tape and crazy glue. And sometimes things are gonna break. And there are so many dependencies down the line. Everybody, you know, everybody listening that could give us some advice on how we could improve also has lots of examples on how something they built wasn't quite reliable all the time because there are so many places where things can break down.
Jon:
Well, there's a lot of moving parts. And, you know, I I think myself a lot of people, myself included, will say, like, well, this is, you know, this is a solved problem. Like, you should be able to not have this happen, but everyone's everyone's situation is so much different that it's it's hard to predict.
Justin:
This is real life. It's it's so easy to from the sidelines, you know, me included to from the sidelines to go, well, why would that ever happen? You know, but once you're in it, you and you're running it every single day. Yeah. So I think let's do a quick what worked, what didn't work, what could we do better?
Jon:
Yep. What didn't work?
Justin:
Yeah. What didn't work? What start with that.
Jon:
We didn't we didn't necessarily have automatic notifications that our main our dashboard was down. Mhmm. Like, we didn't necessarily know right as it happened. There's certainly checks we could have in place that would say, hey. You're you know?
Jon:
Maybe we put checks in place that say that ask our server when our certificate's renewing
Justin:
Mhmm.
Jon:
Or expiring and and, like, warn us ahead of time if it has not been renewed yet. Mhmm. That or we buy a certificate. We actually don't use Let's Encrypt for our main domain. Yeah.
Jon:
And that if something happens with renewal, it's only for the custom domains?
Justin:
Yeah. For our customers, you mean.
Jon:
Yeah. Mhmm. So I think, like, maybe putting too much putting too much, I don't know, like, reliance on things automatically working. Mhmm.
Justin:
Yeah. Not enough, redundancy.
Jon:
Right. Right.
Justin:
I mean, in terms of what worked, people told us right away. Yeah. People told us right away, and on on lots of channels. And I think this is the advantage of us being a platform that serves creators, people creating podcasts, but also serves consumers, people consuming podcasts. Because we heard from I I think we actually heard from the consumers first.
Justin:
We heard from people that listen to shows hosted on Transistor that couldn't get their favorite show.
Jon:
Yeah.
Justin:
And so that was that was nice. And that is probably just a feature of the ecosystem we're in. We are gonna hear from people pretty quickly, on SMS, Twitter, you know, email. Like, I got I got it coming in everywhere. So, that that worked.
Justin:
I also think that, status dot transistor dot FM, that site that I built, it's just a static site, but being able to update things and just have a central place to to send people. So people would email in, hey, is your stuff down? I'd be like, yes. I'm so sorry about this. Here's what's happening.
Justin:
Just go to status. Transistor.fm for updates. And then I could pin that to the top of our Twitter. It was just very helpful for clearly communicating. This is where we're at.
Justin:
This is what's down. And then a longer explanation of kind of what happened. Oh, the other thing that was nice is so we have status.transistor.fm on GitHub pages. So it was up. Marketing site is on flywheel and it was up.
Justin:
And so I was able to put a banner on our marketing site that just said, for users trying to log in, are we're doing some SSL maintenance right now. You won't be able to access the dashboard. Here's the details, and they were able to click on that.
Jon:
Yeah. There's also, you know, along with, I guess, what didn't work back to that and and, like, not enough redundancy is is there are there are ways with some extra work and some engineering to sort of separate out. I kind of I kind of thought ahead with this and and made the RSS feeds their own subdomain and our our, like, media, URLs, their own subdomain so that you can actually eventually split those off into their own things.
Justin:
Mhmm.
Jon:
Their own services. So if, like, the dashboard goes down, people can still the RSS feeds will still work, and audio files will still work. Mhmm. So that is you know, that's a that's a down the road kind of thing where we can kinda siphon some of those off into their own services and make sure they're like, okay. So, like, feeds feeds is now running on its own thing with with its own self certificate that doesn't expire for 2 years or whatever.
Justin:
And Yeah. Yeah. Yeah. You've got good bones there. We just needed to if we had if we had separate SSL, I mean, maybe that's somewhere where we use, like, Cloudflare certs is on RSS feeds and media URLs.
Jon:
Yeah.
Justin:
Yeah. In terms of what we could do better, what what are some of your ideas there?
Jon:
Well, I mean, obviously, internally have some better, like, early warning systems for things that for things like SSL certificates expiring. Mhmm. Because we are depending on auto renewal, we should be able to I think we can certainly programmatically check things and and, like, email ourselves Mhmm. If if they're gonna expire without, being renewed.
Justin:
Another one I was thinking about was, I think when we're small like we are right now, it doesn't make sense for us to hire a full time DevOps person. But maybe we need to hire, like, Joe or somebody as a backup. So Mhmm. If we if we need somebody and I'm like, the so let's say, you know, we have some automated messages that message us. But we also have like, if it's if I'm there, and I'm awake, but you're gone, I I can try to contact you.
Justin:
If I can't get a hold of you, I can contact Joe.
Jon:
Yeah.
Justin:
And we just pay him hourly. Right. For because that would be awesome.
Jon:
That'd be great. Yeah. If I'm if I'm on my 2 week silent yoga retreat.
Justin:
Yeah.
Jon:
And,
Justin:
If you're, like, if you're high on alahuasca or whatever.
Jon:
I'm in another dimensional plane, plane of existence.
Justin:
Yeah. We there's someone else to to to call. I I And, I mean, certainly, I think I could get more of that stuff too. Like, there's no reason that I can't learn some of that. But a a a probably a stronger fix is to have 1 or 2 contractors.
Jon:
Yeah.
Justin:
That, if I can't get a hold of you, I can just contact them. And Yeah. You know, that then right away, we have people that know our setup and can go, okay. Yeah. I know.
Justin:
And Yep. Charges hourly.
Jon:
Yeah. I mean, another thing I think I kinda joked about with you over text messaging is, like I think I said something like, oh, man. I need a I need a MacBook that's the size of my phone. Yeah. So I can just, like, open it up and fold it and, like, type stuff.
Jon:
Because I I I couldn't really there's nothing I could do from my phone, which is another thing, like Yeah. That we've that we've talked about in other episodes is, like, maybe there's certain things that we should be able to do from our phones. Like, in this in this case, would have been a little tough, but
Justin:
Yeah.
Jon:
I don't know. Maybe there's a way to, like, force force renew a certificate or, like, restart servers from your phone or Yeah. You know, stuff like that. So
Justin:
Yeah. No. Yeah. I think that's a good idea because invariably it always happens like that. How many times dear listener, if you're listening, how many times have you been out like, with friends out for drinks or hiking or something, and something comes in on your phone, you're like, oh, shit.
Justin:
I how am I gonna do this from my phone? And Yeah. Maybe what we need is on our admin side, we just need a few buttons that have some pre scripted things that we know we might need one day.
Jon:
Yep.
Justin:
These phones are great. Like, it's more computing power than they had when they landed people on the moon.
Jon:
Oh, way more. I mean, it's more computer power than I had on my first computer But the by far.
Justin:
But they Yeah. Suck for getting things done.
Jon:
Yeah.
Justin:
They they're so bad.
Jon:
I mean, I could, you know, I can I can install an SSH app on my phone, and I could probably log into our servers and do some stuff, but, like, I'm doing it from an iPhone keyboard?
Justin:
Yeah. No. It's terrible.
Jon:
It's awful.
Justin:
There needs to be that would actually be if someone has like an Amazon, gift pack or something, that's just like a little external monitor for a phone and a keyboard. And it just like folds into a little thing and it's just like, you could just break it out and go, okay, now I've got actual, like the emergency pack, you know?
Jon:
Oh, like a rollable keyboard and a rollable screen.
Justin:
Oh, yeah. I mean, do you remember when the Palm came out? The Palm pilot was my college computer because I had a foldable keyboard. And I would just, like, fold it out and, like, type all my notes on this palm. It was it was amazing.
Justin:
Yeah. I could type. So the the palm was so slow. I would type the whole lecture and then I would have to wait like 10 minutes after the lecture was done for the screen to catch.
Jon:
Yeah. I mean, you can you can hook a Bluetooth keyboard up to your iPhone.
Justin:
It's true.
Jon:
It's just that who's gonna I'm not gonna carry it with
Justin:
I know. I know. That's I'm just thinking, like, laptop is laptop's still big. Even if you get a small one, it's it's like and there's also this feeling. Again, there's tension here too because we we don't wanna handcuff ourselves and have to bring a laptop, like, when we're out on a date or when we're, like, getting, you know, going on a family retreat where we're Right.
Justin:
That's a that's a bad life to be had.
Jon:
Yeah. That's that that is a big downside of a small company like this
Justin:
Mhmm.
Jon:
Is that it's sort of unavoidable.
Justin:
Yeah.
Jon:
For a while.
Justin:
Yeah. I like that idea of hiring some contractors on hire though, because Yeah. If we're gone, it's so easy to just text them and say, hey, just so you know, I'm going, you know, away from keyboard for whatever. And to say, hey, you're gonna be around. And then they just know, okay, I might get a call.
Justin:
And if I do, I know what to do. You know? Yeah. That that seems like a a really great solution to, you know, not being able to hire a full DevOps person. Just have Yeah.
Jon:
But yeah. Just to alleviate a little bit of stress.
Justin:
Yeah. Totally. So yeah. That was would you wanna say anything else about that?
Jon:
No. I'm pretty tired today.
Justin:
Well, let me tell you about our second sponsor. Today's episode is also brought to you by Balsamiq. This is a company I really look up to. If Transistor ever becomes a tenth of what Balsamiq has achieved, we would be very happy. Here's what's amazing about them.
Justin:
They've they've done really well. They're bootstrapped. They're extremely profitable. And this allows them to do things that other VC driven companies can't do. For example, every year, they budget This is incredible to be $350,000 for sponsoring events.
Jon:
Wow.
Justin:
They just like giving back to the user experience community that they serve, the entrepreneurial community. So right now, this instant, MicroConf is going on in Las Vegas. Well, they sponsored that. And I've seen them sponsor other kind of design and, user experience UI conferences. Also, they give 3% of their profits to charity each year.
Justin:
And their employees get to choose which causes they support. So they're giving to schools, nonprofits. It's really, really cool. You can see all of what they're doing when you go to balsamic dotcom/givingback. And you might even qualify for something on that page, by the way.
Justin:
If you are running a charity or if you have an event or, you know, you'd like them to sponsor something balsamic dotcom/givingback. Alright, so, one thing I wanted to talk about last week that we didn't have time for is where we get a lot of our revenue from right now. And in the in the, aim of being transparent and just telling people about our journey, I thought this would be an interesting topic. Some people don't know, but we currently have an affiliate program. And so if you have a newsletter, a high traffic blog, maybe a big social media following, maybe you're an agency and you're often recommending tools to customers.
Justin:
You can refer folks to Transistor with your own affiliate link. And if people sign up for a paid plan with that affiliate link, we give 25% of that revenue forever, as long as that customer is paying us. And, I thought this would be interesting because, there's not very many people in our industry that are doing this. Some folks will give you a Amazon gift card. If you refer somebody, some folks will give you a one time payment.
Justin:
If you refer somebody we're doing 25% of revenue ongoing. And that means, for example, if you're following our transparent, benchmarks on transistor.afam/bearmetrics.com, you'll see that we are getting very close to $10,000 recurring monthly recurring revenue. We're at 9,529. It's awesome. It's really exciting.
Justin:
We're we're aiming for that 10 k milestone, But that's not the whole story. And, it's not 25% of that number is affiliates, but a big chunk of that number is affiliate revenue. And. Yeah, I thought it would be interesting to talk about it. Yeah.
Justin:
So I'm trying to, I'm trying to figure out the last time I calculated this, I think 28% of our revenue so far has come from affiliates. I think that's right. Let me I'm just gonna log in here. We're using Get rewardful for running our affiliate program. Can you explain a little bit about how it works with Stripe?
Justin:
Because all I know is that it's better than what I've used before. It just it seems really simple. But how does it work with Stripe?
Jon:
With rewardful? Yeah. So, yeah, it's it is a really simple integration. When you give out your affiliate link Yep. Our sign up page knows, that it's tied to your affiliate account.
Jon:
So when that person signs up, there is a little bit of metadata that gets you can attach to a Stripe account. So it just says, like, I don't know, rewardful account, and then the value is the name of your rewardful account, I think, something like that, or the ID of your user. Yep. And then our Stripe account is hooked up to rewardful. So rewardful actually gets, webhooks for successful payments.
Jon:
Mhmm. So when, when there's a payment on Stripe that goes through successfully, we get webhooks and so does rewardful. And if a rewardful gets a payment for an account that is that was signed up for through an affiliate link Yep. They basically say, alright. 25% of this or however we have it set up at the time, 25% of this, is a commission that you owe so and so.
Jon:
Yeah. And then it just sort of, like, adds their account up.
Justin:
Yeah. And the the cool thing is that, like, I can I'm looking in Stripe right now, for example, at a customer that just signed up. And in the metadata section of that customer, I can see their transistor user ID, which is what we add. But then I can see the referral ID, who the the affiliate is, what campaign they sign up through, what link they used. And then I can view that, that whole thing in rewardful.
Justin:
So I can see, you know, how much have we paid out for this particular referral. And after the fact, this happens all the time. If somebody says, hey. I referred this customer, but I forgot to give them my link. We can go in and edit that metadata manually in Stripe and give that affiliate credit for that sign up.
Justin:
And these are things that in other affiliate programs I've used, you just can't do. So it's, it's really great. And it, it only works with Stripe. So it's really clean. It just, it just works, which is the helpful part.
Justin:
So here's how much has happened so far. Total volume, we've done about $55,000 in revenue. 6 about 16,000 of that is referral income. So that's about 29% of total revenue of gross revenue has come through our affiliates. And the inspiration for this was, this is something I saw work with my friend Nathan Barry, who runs ConvertKit.
Justin:
They really, you know, he started ConvertKit 5 or 6 years ago. And it kind of petered along at, you know, went up to $5,000 a month in recurring revenue, but then went down to 1500 or something. And it just wasn't growing. It was kind of floundering. And there's a number of things that he did that changed it.
Justin:
But I think the the main one was he started to leverage affiliates. He started to say, you know, if you are already recommending, you know, email service providers, recommend ConvertKit, and you will get a share of the income forever. And, you know, if you, if you're already recommending tools, that's quite attractive. Right?
Jon:
Mhmm.
Justin:
And so seeing this work for Nathan, I thought this might be something we wanna look into. But I kinda just left it. I was like, ah, whatever. We'll we'll we'll get to it when we get to it. And Kyle Fox was building rewardful, and he just kept bugging me.
Justin:
Like, Justin, can you ask John if you guys are gonna, you know, implement this? And I was like, I don't know. Okay. And then Kyle would like bug me and ask to do like a, you know, screen share so he could show me things. And Kyle has been a friend for a long time.
Justin:
It shows you kind of how tenacious you have to be sometimes. Because he had Yeah. He really had to work.
Jon:
He was. And it was actually it was really easy to set up. Mhmm. I think we we set it up, and I sort of I forgot about it. I honestly didn't re I didn't realize, like, how much it was being used
Justin:
Yeah.
Jon:
Or who like, who was using it
Justin:
Yeah. In some necessarily. In some ways, me too. And, just like a lot of things, it's I don't know if this is the 80 20 rule. This might be more like I think with affiliates, it's like there's the 99%, and then there's the 1% or the 90 10 rule maybe.
Justin:
There's a few affiliates that drive almost all of this affiliate revenue, because most folks just don't have the audience. Right?
Jon:
Right.
Justin:
So I know, for example, for Nathan, Pat Flynn, who has a huge blog and a huge podcast and has tons of folks, You know, Pat saw ConvertKit, really liked it, said this is way easier to use than Mailchimp. I'm gonna start using it. And then he said, well, I'd like to start recommending this to my audience. And Nathan said, well, if you recommend it to your audience, I will give you a share of you know, I'll give you commission. And that drove a a ton of sales for ConvertKit because he because Pat has a big audience.
Justin:
Likewise, we have a few affiliates that drive most of our affiliate sales. And, you you know, some of these people just heard about it through other folks. It was just word-of-mouth. I because I shared this with just a few people and they told other people about it. And that's how the word kind of spread.
Justin:
Yeah.
Jon:
That's the thing. Like, I don't remember either of us really publicly announcing an affiliate program. It's not really on our site.
Justin:
Yeah. There's a page. Like, if you go to transistor.faml/affiliates.
Jon:
There you go.
Justin:
But it's not like that. It's not like that page is getting a ton of traffic. I think I've announced it a few times on Twitter, But it's yeah. It's not something that we've we pushed super hard. It was mostly,
Jon:
I think, word-of-mouth. Which is kind of amazing. Mhmm. Again, how much how much revenue is coming from it. Yeah.
Jon:
How do you how do you feel about it now when you look at the numbers? Like Yeah. Like, for me, I I'm totally okay with it. I don't necessarily think the people that have signed up and stuck with us would have otherwise Mhmm. Would have found out about us.
Justin:
Yeah.
Jon:
Although, I don't know if you turned these on or if Kyle just started the thing where we actually now we get emails. I just started getting emails from rewardful to my email at transistor. I don't know if you get them too, but it's like, there's a new commission for so and so.
Justin:
Oh, yeah. I just turned
Jon:
It actually I just turned those off.
Justin:
I just turned that off for myself. I think you Okay.
Jon:
Yeah. See, I did I think Kyle maybe Kyle just rolled that out. Yeah. And I actually don't like seeing it.
Justin:
Yeah. You you can manually turn it off now.
Jon:
Okay. Yeah. I saw that, but, like, I haven't yet. But it's just like, I'd rather not know. Yeah.
Jon:
Like, I'm fine with the commission. I'm fine with the affiliate program and, like, giving commission, but, like
Justin:
Yeah.
Jon:
I I was totally fine not knowing how much money we were giving out.
Justin:
Yeah. I mean, there is kind of two sides to this. I was actually I was talking to my wife about this, and she was like, what? You give how much? Because it's forever.
Justin:
And so there's there's this there's 2 sides of this. On one hand, it's like, wow, that's a big chunk of revenue that you're giving up. On the other hand, that it there's gotta be, a way for us to compete. And the one way for one way for us to, you know, get more folks signing up that wouldn't normally hear about us. To to be honest, I'm surprised there's a ceiling to how many people from your personal audience and your personal network will sign
Jon:
that. Right.
Justin:
And when you're brand new and you're still trying to build up, you know, your search engine optimization. And we haven't even started advertising yet. And, you know, word-of-mouth is going to, you know, play a piece in this, but word-of-mouth is amplified as you get more customers. So as you get more customers, they tell their friends and, you know, you get some of that rolling. But in the beginning, you need channels.
Justin:
And, really, startups and products live and die by efficient channels. You gotta have efficient ways of getting to the customers.
Jon:
Yeah. It is. I I think it's still you mentioned advertising. I think it is a really inexpensive way to advertise because you're not like, with most advertising, you spend the money and hope you get the customers, but with affiliate programs, you you you're spending the money after you have the customers.
Justin:
That's right. Yeah. You you know So you're gonna get them. I I think there's also this this thing. What one thing that really worked for Nathan and ConvertKit, because it was the same thing, MailChimp.
Justin:
They have an affiliate program, and everybody says it sucks. So and I should say people that kind of refer things, or are are big in the affiliate space, they're all like, the Mailchimp affiliate program. It's brutal. Right?
Jon:
Although there's like yeah. It's just fun. I don't know why it's
Justin:
fucking weird.
Jon:
It's just like
Justin:
I I I realized that some people think this affiliate thing is yucky. And I'm, I'm trying not to dance around it too much. There are people who, you know, professional bloggers, influencers, quote unquote, you know, there's people who recommend products for a living. And I think just like anyone, in a sense, it's like it it's very similar to advertising. So when Marco Arment takes advertising money for his podcast and then recommends a product, that's like, a upfront affiliate payment.
Justin:
They're saying
Jason:
Mhmm.
Justin:
We are giving you this money hoping that you will give us a return on this ad. Affiliate is a is a a back so that's a front end payment. Affiliates is a back end payment. If you refer us refer paying customers to us, then we pay you a commission just like we you would a salesperson. Yeah.
Jon:
And Yeah. I mean, advertising has changed so much on the Internet in the past 10 years that people used to make money, you know, with Google AdWords. Mhmm. They don't you know, they're not making anything on that anymore.
Justin:
Yes.
Jon:
They're still providing good content. So
Justin:
Yeah. Yeah. This is another way for independent, like, a few of our biggest affiliates are just independent blogs that are blogging about the podcast space. And so they they make their living, writing about, you know, podcasting and podcast news and things like that. And then on as a part of that, they're recommending products.
Justin:
And you legally, in the United States at least, you have to, say that you're these are affiliate links. You have to say, you know, these are promotional links. And what most folks say is, you know, these are products I would be recommending anyway. But when you use my link, it lets them know that I sent you. And, in in, I get paid for that.
Justin:
I get I get some compensation for that. But it's actually very fit. Podcasting advertising was actually built on this model. Right? Like for a long time, I I whenever someone asked me, you know, hey, I need to sign up for hover or whatever.
Justin:
I always said, oh, just use the code Dan Benjamin. Because that was Dan Benjamin's 5 by 5 code.
Jon:
Yeah.
Justin:
Well, that's a that's a affiliate link. Every time someone signed up for a domain on hover, he got, some affiliate money, right? So there's two sides of this. On one hand, I go, oh, man. That is a lot.
Justin:
That gross 25% of that revenue is being paid out to affiliates. And that, man, that that's feels like a lot of money.
Jon:
It does. But I have a feeling that we would be our revenue would be much lower than that had we not
Justin:
had it. Definitely. And so, it just gives people another incentive. Now this isn't a good it's this doesn't work if you're just casually recommending transistor to your friends, which you should do anyway. Like casual kind of recommendations, it doesn't work well for those folks generally don't end up earning any kind of income from it.
Justin:
This only works for folks that have a big audience, like a big podcast, where they can recommend, you know, something and have a lot of people respond. And so, to access that audience, we could pay upfront through an ad or we could pay on the back end with affiliates. And it would actually it'll be interesting for us to compare those 2. Maybe it is more efficient to actually just pay for an ad.
Jon:
Yeah. It's not really something we've tried yet.
Justin:
And and that's something actually I'd like to track through rewardful. You can create these campaigns. So, you know, in this affiliate campaign, we have 25% commission, but I could create another campaign that's called ads that gives 0% commission. But I you just use the links for tracking.
Jon:
Yep. Yeah.
Justin:
And so we'll be able to track that and go, okay, well, actually this is much more, this is way more efficient for us to to to do ads or whatever.
Jon:
Or if if you become dis disillusioned and dis, you know, upset about how much how much we're paying you, you could set up your own affiliate program and not tell
Justin:
me about
Jon:
it And then pay yourself
Justin:
a bunch of money. That's right. I'll just I'll just, exactly. Yeah. This is a way of me siphoning off money from the company.
Justin:
Oh, and the other thing right now is we can only we have to pay out affiliates manually. And so I only do it when people have reached $100 So if you are an affiliate and you haven't got a payment yet, it's because you haven't reached $100 in accrued commissions yet.
Jon:
Yeah. And we could certainly automate that. I believe, Kyle has an API with Yeah.
Justin:
They're they're they're working on it. That's that's like
Jon:
We could we can certainly automate that.
Justin:
Eventually, I think
Jon:
Through PayPal
Justin:
or something. Eventually, we're gonna that will all be automated. I'm sure people are gonna have questions about this because this is the number one thing people ask me is, how are you guys getting customers? And, this has definitely been a significant channel. Like, we've we've gotten a lot of paying customers from affiliates.
Justin:
And they don't all stick around. But, you know, even in our chat widget, when people ask for help, I always ask, hey, so how did you hear about us? And often it's like, oh, I was like reading, you know, reading this guy's blog and he recommended you along with some other ones. So I checked you out and I really liked your player. So I chose you.
Justin:
It it it it can be depending on what kind of product you have. And I know, for example, Basecamp tried affiliates for a while, and it really didn't work for them. If you're interested in trying this channel, it might make sense to, to check it out. And, of course, there's other ways of doing it too. You don't have to have as high of a commission.
Justin:
You could do one time. Although I don't know if rewardful supports that yet. But, yeah, there's there's other ways to run it. I I think that's what I was gonna say is that I part of me feels good about this because we're really, benefiting from these audiences that people have built for a long time. And just like I want creators to get paid for the work that they do, I want to pay these creators, these people that are are making content or recording podcasts or whatever.
Justin:
And if, be becoming a transistor affiliate helps them to make an independent living. I'm all game for that. I think that's great. And, 25% feels generous. Like it feels like this is we're we're really, you know, we're we're trying to take care of the people that are helping us.
Justin:
So yeah. Anything else you wanna say about that?
Jon:
Not really. Other than I'm, you know, pleasantly surprised by how well it works.
Justin:
Yeah. And it's also funny how how it was kind of accidental. Yeah. So well, let's, let's run through our Patreon supporters. We've got we've got a couple new ones.
Jon:
We we do. Yeah. Let's see here. As always, thanks, to our supporters on Patreon for for helping out with the show here and helping us pay for our editing costs. We have a few a couple new ones here.
Jon:
Michael Sitberg. Yeah. Do you know I don't
Justin:
know Michael. Actually, I might know Michael. I I do know Paul Jarvis and Jack Ellis. They they supported on behalf of Fathom Analytics.
Jon:
That Okay.
Justin:
Some people were figuring out how to get a little, like, commercial shout out here at the end of there.
Jon:
Yeah. Yeah. Well, they're gaming the system. It's fine. My brother Dan Buddha, at Daniel Buddha on Twitter, danbudda.com.
Jon:
Darby Frey, Samori Augusto, Dave Young, Brad from Canada, Kevin Markham, Sammy Schuichert, Dan Erickson, Mike Walker, Adam Devander, Dave Yeah. Kyle Fox from get rewardful.com, which we were just talking about, and our other sponsors, Clubhouse and Balsamiq.
Justin:
Yes. Thanks, everyone, and we will see you next week.
