Hosted by Scott Cohen and Garin Hobbs - Talking about the hot topics in and around email marketing.
Hello all. Welcome to that inbox army podcast. I'm your host Scott Cohen rocking that wasn't very data driven of you. Shirt today, very appropriate for our discussion today. And with me today, the Magic Nan to my El Diablo is my co host, Garin Hobbs.
Scott Cohen:Garin, how are you doing today?
Garin Hobbs:Nice, Phoebe. Now you don't, Scott.
Scott Cohen:Shaking bake. Alright. Today, we're gonna tackle testing. I've said it before and I'll say it again. The best email marketers are really great testers at heart.
Scott Cohen:They're always seeking out ways to find sufficient data to prove an ongoing list of hypotheses about how their audiences will engage with email. It's more than testing to say you're testing you're going you're testing to learn something but setting up tests properly can be tricky to help us build the case for testing and how to accomplish this testing in the right way Joining us today is the ceo of holistic email marketing and best selling author of the book of the same name and someone I'm thankful to be able to call a friend Kath Pay. Kath, welcome to the podcast.
Kath Pay:Well, thank you both for, very much for having me on. I'm very, very excited to be here.
Scott Cohen:Yeah. We're excited to have you. Before we get too deep in the weeds, I love to learn about people's journeys and stories, how they ended up to where they are now. So tell us about your journey to today.
Kath Pay:Okay. I'll try to give you the 60 60 seconds spiel of it because otherwise but, essentially, I was, I had my own web design company, and we're talking about in the mid 19 19 nineties. Okay? And, we had some great great customers, and my head programmer and I, created this nucleus of an email, service provider product for one of our clients who had a need to do one of these new things. What a email marketing, I think it is.
Kath Pay:Oh, okay. I'm sure we can do that. Right? So we did that. And then pretty so soon we, you know, had all of our clients on it or everyone's using it, and it took over my web design agency.
Kath Pay:So before you knew it, I was an email marketer, and that's what I had. I had an email marketing system. So 1998 was when I officially became, yeah, an email marketer.
Scott Cohen:We're we keep getting these OGs on the on the podcast here, Garin. We had, you know, Matthew Dunn previously. He what what was the date he dropped? 93, 94, something like that? And
Garin Hobbs:I was still in high school. That's for sure. But, it's I love that I love that story, Kath. I mean, my my my my journey has been rather similar. I began an email in in 1999, and what a wonderful, unique, and humbling experience to have the trajectory of your career match almost exactly the arc of the trajectory of a new and emerging industry.
Garin Hobbs:Right? I just feel very fortunate to have been here at well, what I what most folks would like to consider the commercial beginnings of email all the way up to today and just seeing the the pace of evolution, the pace of innovation, what has changed, and more importantly, and I think more excitingly, what hasn't changed. Right? So I love that, I love that we still have folks like yourself and myself in here just kind of bearing the torch and keeping the industry marching forward.
Scott Cohen:Yeah. I'm the baby on this call.
Garin Hobbs:Awesome. Well, getting back to testing, Kath, it's it's it's always struck me that, AB testing, very similar to scientific method. Right? Hypothesize, set the experiment, measure, conclude. Would you say that's a fair comparison?
Garin Hobbs:Or if not, how would you say AB testing is different or departs from scientific method?
Kath Pay:No. It it we literally should be using the scientific method. And we have, at Holistic, we have developed the holistic testing methodology, which is based on the scientific method. It's all too often I find, and this is where a lot of the problem comes when it, with email marketing and AB testing, there's a lot of, a lot of information that isn't well known within the industry. And, but, and I'm not blaming anyone in particular, but that's just a fact.
Kath Pay:So what that means is that most of us have fallen into email marketing. Most of us don't have marketing degrees. Most of us don't have a behavioral science degree. Most of us, you know what I mean? Like we we're not mathematicians and everything like that.
Kath Pay:So we don't necessarily, we haven't been taught the scientific method. So what that means then is that we tend to be using the, the tool that we have to hand, which is our email service provider, our ESP. And we tend to think that that's what you do. That's how you test it, but they're, they are flawed. So they're not used they're not following the scientific method in any way.
Kath Pay:They're doing a very shortcut, a very, ad hoc approach to testing. And hence email marketers nowadays are literally taught what I would say, and and I have been on record saying this before, the wrong way to test. And because of that, what happens is that that makes everyone become very disillusioned because the results don't stand up because they're they're not statistically significant. They're not using a hypothesis. They're doing a test only on that one test rather than doing multiple tests to prove the one hypothesis, you know, and therefore just discount anomalies and all the rest of it.
Kath Pay:So that's where this disillusionment settles in and, oh, yeah. We tried testing. Yeah. It didn't work for us. And then that's it.
Kath Pay:You know, so it's, it's sad. So I am on this mission to try and, you know, educate people so that we can do testing better because it has so much more to deliver. You know, we talk about it in theory. I've got some amazing case studies that I do with my clients and we can get fantastic results. You know, the other day I've got a 17% uplift in conversions, and this is of an ongoing running, automation program.
Kath Pay:17% in conversions for that. That's actually quite revolutionary when it comes to, you know, an ongoing program like that. So, you know, it really can make a difference.
Scott Cohen:Let's dive into statistical significance a bit here. I don't think people fully understand what that there's an there's an equation to it that I've had data people try to explain and I go, I don't understand how a 1.58% versus 1.57% is statistically significant, but a 4% difference isn't. And I I I don't need to we don't need to go completely nerd dom there, but it is a big necessity because testing needs to amount to even if test results are flat, you need to amount to where you go. They're flat, and we tested it enough to know that what are those guardrails what do those guardrails look like for you like volume you talked about you got 17% conversion uplift like what repeatability you you mentioned people just test it once and you go, okay, well, that's what you learned on that Tuesday at 9 AM. You know, walk us through a bit more of that, you know, that process that you have and what are those necessities that you need to make sure you get stat sig?
Kath Pay:Sure. So as far as the statistical significance goes, I, too, I pull up my hand. I am not a mathematician. Right? I actually say in my book, But now, you know, I love it and I embrace it.
Kath Pay:I can't think of going a day without doing math. However, calculating statistical significance is another depth that I am not prepared to go. But luckily for us, there are all these amazing free calculators out there who have been created by much cleverer people than us or than me. Okay. So and I rely upon them.
Kath Pay:So there are ones that can let you know that this test you tested a against b and it's going to and then you put the the total that you sent it to, what the uplift, you know, so what the percentage was, whether it's opens, clicks, conversions, for for each of those. And then it will tell you whether it's, statistically significant. That's what I use it for, and it's fantastic. Then there's another one that you can use where you actually have to work. You you can use it to calculate the necessary sample size.
Kath Pay:Now I personally don't use that. And the reason why is because of that repeatability. Okay. So what I wanna do is I want to make sure that, most of the tests that we do are, are generally a 5050 split test. And the reason why we have them as a 5050 split test is because I don't wanna test open rates.
Kath Pay:When it comes down to deliverability, if I, if I have a client who is struggling with deliverability, then yes, I'm gonna be testing subject lines using the open rate because that is the meaningful metric for the ISPs as far as deliverability goes. But beyond that, no, I'm wanting to be looking at other things. So click through rate, if at all possible, conversion rate. So it's all gonna be dependent upon what the objective of the, of the email is. Now having said that, if you're a publisher and you're being rewarded on opens, then of course that's gonna be your key metric.
Kath Pay:So you have to, you know, take each business into account. But for a lot of the retailers that I work with, that's not their key metric. They're not rewarded on opens. They're actually rewarded on conversions. So when we talk talk about using conversions as a success metric, what we need to do is understand that it's not gonna happen within 2 hours.
Kath Pay:Right? Those conversions will come over a period of time, 1 week, possibly 2 weeks. So we firstly need to understand what is our general, and I've written a lot about the long tail of email marketing as well. And that's basically when you go and you keep on getting more and more conversions and then it kind of drops off and then you start getting less conversions. Right.
Kath Pay:But they still don't stop that one campaign.
Scott Cohen:Mhmm.
Kath Pay:4 weeks, 6 weeks later is still delivering conversion. K. Amazing. I'm not asking you to wait 6 weeks. I'm asking you to have a look and determine what is the ideal time for you to be measuring what that conversion would be.
Kath Pay:Often it will be between 1 2 weeks, 1, 1 week, 7 days, 10 days, whatever the story is. So have a look at that. And then when you look at that, you realize you can't do a 10, 10, 80 split. You realize you have to do a 50, 50 split, let it run and then report on it. And do you know what I would do for the first couple of ones that you're doing, keep going back and, and updating it and see if you chose the right period of time.
Kath Pay:And if you've so be particularly like, you know, maybe a one at 7 days and after 14 days B one, Right? Then you realize, oh, maybe I haven't chosen the right period of time. Mhmm. You need to these are things that you guys have to sort out for yourselves and test out for yourselves and do a little bit of, you know, research for yourself, because every product brand company audience is different. So that is the reason why I don't that's a very long story.
Kath Pay:Why I don't use that statistical, you know, calculator as such. It is there for you to use if you do wanna be doing it, particularly if you've got going to be doing, you know, open rates as a success metric, because you can use the 8080 the the 10, 10, 80%, you know, test for for open rates.
Scott Cohen:But I think that you you call out the time frame really well, though. Right? Because in in previous life, if you're doing a subject line test, you're doing 1010 80. Some people go wait an hour. And I go, no.
Scott Cohen:Here's the problem. The number of times we did that and then the selected winner did not ultimately become the winner from that 1010 group. I mean, that's the danger that you have. I mean, fortunately, in those scenarios, the difference wasn't so much that I felt like we were losing out on so some things. But for some things, I have yet to see a subject line do that much better where it wasn't something else at play.
Scott Cohen:Right? Like, it's just so that timing, that that setup, that long term, that repeatability, I mean, you know, I've always said be careful about looking at Tuesday at 9 AM because you're only learning about Tuesday at 9 AM is the same thing carrying across multiple days, multiple audiences, or maybe not multiple audiences, but, you know, that's where the learning comes in because that moment in time, who knows why nobody opened that day, or this many people did and this many people didn't for that one day, that one snapshot. Right?
Kath Pay:Yes. And going back to what you're saying about the, you know, the the one hour window. Right? Nowadays, with the number of box that we have on our list, then, we obviously would have hopefully have some very loyal followers who are gonna be opening up every single email. Now what about if you actually had inadvertently put a high percentage of those devoted followers, and one, on the variant a, but not on the variant a, but not on the variant b.
Kath Pay:Right? So you can't they're the ones that are gonna be opening up soon, straight as soon as you send it. So you have to be very, very, careful, very, you know, thinking everything out, planning, testing, observing like you did. But then when it comes down to the hypothesis, and this is the crux of the matter, If you use a hypothesis, what that means then is that you can be testing that same subject line hypothesis multiple times, but you're gonna use different wording. We at holistic within the test, the holistic testing methodology, we take it one step further because we really like to be testing, motivations.
Kath Pay:So curiosity versus, loss aversion, persuasion, even one of the tests I've done recently is just using persuasive language, versus just the control, which is very perfunctory language. And so that is a, a, an amazing lesson. You can look at and go, oh, wow. It does, it works. So what you need to do is you need to have that hypothesis in place, and then that allows you to use different copy every single time.
Kath Pay:Because you are supporting and following that hypothesis. And then over time, if you get a 2 out of 3, say 2 for B and, and 1 for a then, okay, you've got a winner. You might wanna test a couple more times just to make absolutely sure, particularly if it's going to be a, a, a big change, you know, if you are, if you've learned something big and then you're gonna implement it across the board, you know, a template or something or different footer, something like that, test it probably a few more times, I would say. But, you know, 2 out of 3, it's good, but it's not, could be better. So that just gives you the ability to test it multiple times without repeating yourself and without boring your audience to death and then going, wait on, didn't I get this last week?
Kath Pay:Right? What's the last thing you wanna do? And the problem is, is that more often than not, we're not using hypotheses and, and that's what's holding us back.
Garin Hobbs:Struggling to hold myself in check here. You said, you know, talked about testing motivations, an area I'm particularly passionate about, and I immediately want to run off in, into that particular tangent, but I'm gonna hold myself back here, and I don't wanna continue to beat the drum on a couple of things that we've talked about here recently. I think this is so important because a couple of things you mentioned just seem to be completely antithetical to the approach that you see a lot of sort of marketers taking today, specifically with regard to, you know, period of time for testing, your time frame for that test. You talk to most marketers. They wanna run to that statistical significance as fast as as they can get to it rather than really holding back and seeing if the truth stands the test of time.
Garin Hobbs:Right? I mean, I think we've all experienced it, you know, in our own roles as marketers. Almost anything new that you try or that you roll out, any new tactic or strategy tends to give you a little bit of lift. But what's the sustainability of that lift?
Scott Cohen:Am I
Garin Hobbs:going to see the same thing if I continue to do this? Is this a snapshot in time, as we said? Is this just that momentary blip where I happen to get everything right, and will it still continue to hold true next week? Right? And that all tends to go out the window.
Garin Hobbs:They're just so happy to see the blip that that becomes the truth, and they immediately sort of run with that as well. Right? So, I'm really glad that you're calling that out. I think that's something I really want people to take away from this is, hey, you know, truth is temporal. Right?
Garin Hobbs:And it's circumstantial. Or at least oftentimes, it can be. So extend your testing period. Continue to apply the same test under different times, different conditions, and see if it holds up.
Kath Pay:Yep. Well said.
Garin Hobbs:So a lot of the people we speak with, Kath, and, you know, we've kind of alluded this, to to this a little bit today. You know, we tend to use testing to chase incremental improvements primarily in revenue. Right? And for the e commerce market, that tends to make good sense. Leading folks like publishers to the side, for publishers, the email is the product.
Garin Hobbs:So leaving those to the side and just focusing primarily on retail e commerce, aside from revenue, what are some other high impact areas to test that are sometimes or even often overlooked?
Kath Pay:Okay. So I mean, revenue is the end goal. Right? So if you're doing frequency, that still affects revenue. If you're doing a change in template design, that still affects revenue.
Kath Pay:It's like everything at the end of the day, particularly if you're ecommerce, that's going to affect that. You can put other KPIs in front of that. You could say number of transactions, average order value, things like that, but that's still going to be around revenue. You can be chasing engagement, and I do have some clients that engagement is very, very important to them. And that would be more though if you're like a SAS provider and the engagement is there more to drive, okay, so they're still relating to you.
Kath Pay:They're still, you know, they've got a, a, a product and they have got a license for that product. And maybe they have they're not logging in as often as they should. You want them to stay engaged. You want them to stay connected with you even if they don't get to that final stage of logging in and using the tool, at least the fact that they're opening, that they're engaging, is showing that there is still some recognition, some interest, and everything. And then fingers crossed, they will go in and log in and hopefully renew their license the next year.
Kath Pay:So, there's, you know, engagement is is one factor, and that's only one example, obviously. But, and then you've got the design. But again, I think it all comes down to, you need to understand what your objective is and more often than not the object. I mean, we did just interrupting myself. We did used to have obscure, what I call them obscure others probably still think they're great ones, objectives of, you know, growing your Instagram audience or something like that.
Kath Pay:I think you're using email. I think that Instagram will grow itself using Instagram, or, you know, you can use another social channel for doing that. You don't necessarily need to be using email. I think email is more, you know, suitable for that true conversion rather than just driving traffic to. But at the same time, having said that, that's what we do is we drive traffic because we are the push channel.
Kath Pay:So we drive traffic to the website. We drive traffic to Facebook, to Instagram, all the rest of it, to the event page because we are that push channel. So may maybe I'm maybe I'm saying something wrong.
Garin Hobbs:No. I don't think so. And and and I love that you you focused on engagement. Is that right? I tend to look at engagement as revenue deferred.
Garin Hobbs:Right? Especially when I think about those sort of high dollar, long tail infrequent purchases, mattresses, automobiles, you know, things of that nature where you buy it once. You're not necessarily gonna be in cycle for a good number of years. But and so what I try to encourage a lot of my customers to remember is that you're not necessarily having to drive conversions. You don't need to convert every customer with every email because it's, a, that's unrealistic, and, b, think about Scott's recent post, you only need to buy so many toilet seats, within a certain stated period of time.
Garin Hobbs:Right? So continuing to market with them is just nuts, and folks shouldn't expect to see wild results out of that. But I tend to look at engagement as keeping brand top of mind, top of consideration for when folks do come back in cycle, they wouldn't necessarily think to go anywhere else other than your brand because you've taken the time to engage with them, deliver hopefully content that is of some value outside of the transaction itself.
Kath Pay:Absolutely. You know, and an example is is, Holistics, newsletter. We have a twice, monthly newsletter. We don't sell our products there. We do have one little ad space there where we might sort of talk about one of our products.
Kath Pay:But other than that, it's just informing, and it's just getting that engagement. It's getting that so that when someone is ready, they're gonna go, oh, yeah. Help her. You know? And and that's what this is for.
Kath Pay:Right? So so it's, that engagement and is all about building that relationship too.
Scott Cohen:Yeah. I mean, it's it's the branding value of email. I mean, I've talked about it before. I mean, when I worked in 1 800 Contacts, we literally sent the same damn emails every week. We had I mean, depending on where you were in the life cycle, you got the same damn email every week.
Scott Cohen:And, Kath, to your point, we had people converting on an email that they got 6 months ago, that they got this that same email that week. And we go, why did you go back and click on the one that we sent you 6 months ago, not the one that we sent you this week? It makes no but people don't make sense. Right? People don't make sense.
Scott Cohen:And that's another key thing I would say. But I I think that there's a lot of the the the timing makes sense, and I think conversion is not necessarily just from conversion rate. It's also customer conversion as well. Right? I mean, you may not get direct attribution.
Scott Cohen:You may not get direct click. I mean, the great example, I think John Caldwell said years ago, and I've borrowed it every time I talk about this is if you get an email saying your bill is due in the subject line, you go pay your bill but never open the email. Did the email work? Yeah. It did.
Scott Cohen:So it's you you gotta look at, you know, the the desired actions whether they engage with it directly or not as well. And I like your really it's it showcases that email is everywhere beyond just that initial conversion as well. You talked about SaaS providers, you know, revenue, you know, not deferral, but, you know, that you're you're building up, you're keeping revenue if you get people to start using that SaaS, right, that that software, those things like that. You're not gonna see ads in Facebook once you sign up for a trial, once you sign up for an account for something, right, emails, the the channel that's gonna drive those things. And if you can get more people to engage and use the platform, theoretically, if they use it, they'll like it, they'll stick around that stickiness factor, email drives that.
Scott Cohen:So that's a great testing ground. I wanna ask this because I mentioned at the beginning, but how do we get people beyond testing for the sake of testing? The old cool hand, Luke, I'm shaking it, boss. You know? Some I I've done this when I was brand side, like, yeah, I'll sure.
Scott Cohen:I'll test. Sure. No problem. And then you just start and really subject lines are where people do a lot of it. Right?
Scott Cohen:So I'm curious because it's easy. Right? It's easy to do. You can show a test. Go, hey.
Scott Cohen:Hey. I shook it, boss. Here's the here are the results of my shaking. How do we get particularly subject line testing to a point where it's actually meaningful?
Kath Pay:Okay. I'm gonna I'm gonna I'm gonna change your question, and I'm gonna go beyond the right, if that's okay. So what when I, when I teach about testing or present about testing, I get a lot of people coming up afterwards and saying, you have inspired me. I am now going to go back. I'm thinking about it differently.
Kath Pay:I'm excited and all the rest of it. And this goes back to my very first point, it's because they didn't know. So what you don't know, you don't know. And they didn't know that they could do it any other way. They're there going, I know.
Kath Pay:Common sense tells me that testing makes sense, that it should give me an uplift, that it should work, but I don't know how to, how to get it. So with our testing, holistic testing methodology, what we do is we're using this, the motivations, we're using the hypothesis and let's take an obvious one that I've already just said. And in fact, we actually ran this test the other day on our list, loss aversion versus curiosity. Okay. So let's have a look at that one.
Kath Pay:Now, are we gonna just test the subject line? No. We're gonna test the copy. We're gonna test the call to action. So we are now in some ways, some people might say, but that's a multivariate test.
Kath Pay:It's not because we're supporting the call to we're supporting the hypothesis. So the subject line, the copy, and the call to action all support the, the curiosity 1 or the subject line, the copy and the call to action, all support the loss aversion 1. So now we've got 2 quite different emails and they, they sound different. They, they all look the same. Obviously they've got the same design and everything, different subject lines, different copy, different call to action.
Kath Pay:Because we've actually tested 3 variables within each email supporting the same hypothesis, we are more likely to have a statistically significant result because we're not just testing 1. We're not putting everything riding on the subject line. And I'm gonna tell you the results of this test. Right? So the results were this is interesting.
Kath Pay:So if you were looking at the subject line and you were basing the subject line, there's a loss aversion versus curiosity. Loss aversion in this test got the highest amount of opens, curiosity didn't. Conversions, curiosity won by mile, even though he had lower amount of opens. How interesting is that? Right now I do this test a lot with generic versus specific subject lines.
Kath Pay:Because everyone always says how short or how long should the subject line be? And I say, well, the question is how generic or how specific should the subject line be? Cause that will decree whether it's short or long. So it's actually, you're testing specificity versus generality. Now general or generic subject lines will give you more often than not, and I'm talking like 99% of the time, will give you higher open rates.
Kath Pay:And that's because what you're doing is you're giving them the ability to read into what it is that they want to get from you. So say you've got 40% off today sale. Fantastic. Oh, man, you're going to get people opening like crazy. Oh, but then it actually turns out it's only of jeans and shoes.
Kath Pay:Whereas I actually wanted a dress. So now I've got there, but I haven't clicked through and I haven't converted because it wasn't what I had read into that subject line. Whereas if you say, you know, 40% off jeans and, and shoes today only, then you're going to attract those people who may have missed it, but you're gonna attract them because they they're actually wanting jeans and shoes and they're more likely then to convert. So I found it really interesting to find out there was a similar type effect happening with the loss aversion and the curiosity. We ran it over 3 tests.
Kath Pay:It was to provide one of our events. And the first one, was wasn't a winner at all. Right? So it was, it was dud. Didn't didn't make any difference.
Kath Pay:The second one loss aversion, 1 in opens second and, and curiosity, 1 in conversions and same same happened with the, with the third one. So it just gives you the ability to have a more powerful, more robust test, but you can't do that with everything. Right? You have to use common sense. And in this case, because it was motivations, we can do that with with multiple things.
Kath Pay:Obviously, if you're only gonna be test if you're testing an image, then no. You can't use it. If you're testing a button, you can't use it. Right? The orange button against the blue button, then you're only testing the button.
Kath Pay:So there are certain limitations. But if you're testing particularly copy, then you can be testing, you know, multiple factors.
Scott Cohen:Yeah. I I would argue if you're if you're testing button colors, I hope it means you've tested everything else because that feels like the end. Right? Like, that's like you're testing the end, not the beginning. Like, oh, button color is gonna make a huge difference.
Scott Cohen:It might, but what about the rest of the test in front of it?
Kath Pay:Yes. Yes.
Garin Hobbs:Very interesting. What so what I heard there is, look, subject lines don't stand alone, Body copy and, creative and your message content doesn't stand alone. These things are an ecosystem. So when we think about testing hypothesis, it's not testing element against element. We're literally testing the efficacy of one ecosystem versus another.
Garin Hobbs:Mhmm. Is that a pretty Yeah. Fair summary or am I completely off my radar?
Kath Pay:No. No. No. It is. We're testing, you know, one unit against another unit.
Kath Pay:And within each of those units, they've all got they've got, you know, different variables. The main thing is to only ensure they're the only variables that were being tested. I did a test for a client recently. They're in charge of the creative. I gave them the brief.
Kath Pay:I gave them the wireframes. I gave them the instructions. They sent it off to, someone else to code it. That person didn't follow through. I didn't get to sign off on it.
Kath Pay:They set it off, and I'm going, why did it fail? This really should not have failed. And I I wasn't just because it was the hypothesis didn't work, but I'm going, there's no logic to this because what it was, same in this particular instance, they both had the same subject line. Right? And it was only the body copy that was different.
Kath Pay:Yet. The control had a higher, something like 10% higher by 10% subject line, open, open rate. Even though they had some going, there's something wrong. It turns out the optics was completely different and a lot of people had images off and, they missed out, some personalization factor and everything like this. So we now in that particular one, that was an invalid test because we weren't actually supporting the hypothesis.
Kath Pay:There were a lot of other factors that were not involved in the hypothesis in the test. So we had to throw that one away.
Scott Cohen:Well, it's like you can test multiple variables because the variables feed up into the hypothesis, but you need to make sure that everything else is the same. Right? Exactly. The the the execution is important here because to your point, if alt text is different, if images show up in one test and not in another, you're you're tying one hand behind your back. And, yes, we say that the two sides need to be different enough that you're actually learning something substantive, but it shouldn't be like that where it's, well, images showed up in one and the other one didn't have images at all, and it's all I mean, unless you're running a design versus plaintext, which great test, by the way, easy to execute, and ugly cells.
Scott Cohen:And I've said that a 1000000 times, but ugly cells. But, yeah, it's that's crucial. Right? Is that you make sure that you eliminate as many of the unintended variables as possible.
Kath Pay:That's it. That's it. So when you're when you're you know, we've now got an an app. I thought that the process was in place, but I made an assumption, which I shouldn't have. So now I've gone back and I've taken complete control.
Kath Pay:Got my magnifying glass out. I'm looking at everything, making sure that they are literally apples with apples, you know, comparing apples with apples except for those variables that we have chosen to support the hypothesis.
Garin Hobbs:Scott mentioned, or uttered the word execution. And I now am thinking about that word in all of its definitions. Winter is coming. Right? And most of us and all of all other marketers are super busy preparing for their our most frenetic and yet most profitable time of year.
Garin Hobbs:When we think when it comes to testing, what should a how should a marketer approach or think about testing during this critical holiday season? What's different? Less the same?
Kath Pay:Yeah, it's, it's incredibly, it's incredibly tempting because often you're actually pulling in different audiences that you don't often nail to. Right. But for me, particularly if you are a retailer, I think you're going to be so, and I have had clients that have gone through and done tests and they've done them successfully within the holiday period. And that's because they had really nailed, well, I had nailed with them the ramp up strategy. Okay?
Kath Pay:So, you know, we've got this great ramp up strategy, and then there was that period when we're just, like, doing this, and and we could put some solid testing in place because we weren't worried about the ramping up or anything like this. Deliverability was a 100%. Everything was amazing. And we could just sit back and just, you know, enjoy the testing and make the most of this very large audience. But at the same time, we have to also recognize that those results aren't going to necessarily be able to be carried through to your non, seasonal, you know, your, your other seasons and stuff, because they are talking about different audiences.
Kath Pay:And this then is talking about when you're testing, test, like I like to be testing segments. So I, I know I've done this before with some clients. We'll test something with, with, you know, the, the newly acquired customers. And then we test the same thing against the, the the the regular, loyal ones, and you're gonna get totally different results. Sure.
Kath Pay:Right? So we have if we're doing that and then we're also testing it during the peak periods when we're emailing a whole heap of different people that we don't normally email, then we have to understand that that can't be, so we're testing in this case, we're testing for the results, not for the learnings.
Garin Hobbs:Yeah. Audience is certainly one aspect of that. Yeah. We do tend to sort of open the database a little bit wider as we go through holidays and reach into the darker corners of it and see what we can eke out of these folks that maybe have been disengaged for some time. But I also think about the landscape, across which this plays out.
Garin Hobbs:Right? We have more competition for mindsharers. We have more crowded inboxes. Every marketer out there is, you know, firing these messages out as fast as they can create them. We have fiercer competition for the same wallet dollar, so to speak.
Garin Hobbs:And customers, prospective customers, buyers, you know, sometimes we're a little bit more free with our spending during this time. Sometimes we're a little more conservative because we've already decided, hey, I want to earmark this portion of my budget for x, y, and z. You know, understanding how that see state, you know, changes very differently during the holiday period, how would that affect, how somebody might approach testing in addition to just understand in addition to the audience impacts themselves?
Kath Pay:Well, that that's it. So we're now talking about financial, worldwide factors, country factors. We're talking about, you know, who's being voted in, any changes in legislation, all of these, they're going to be affecting what happens. So we have to take all of that into account. So this is when we're doing our reporting.
Kath Pay:We might wanna say, actually, this was 2 days before the general election or something like this. Right? And with a little note, test this again later on. So yeah. So we we do and every year, we have that post, holiday season analysis, don't we?
Kath Pay:Well, it was different this year because of this and that and all the rest of it. And everyone had their wallets tightened. They were only going to be buying what they had already planned to buy and blah, blah, blah, and all the rest of it. So we have to take all of that. We need to be doing our own analysis, and we need to be reading everything else that we can find that everyone else is finding as well in order to be feeding into our own analysis of what it is and why why it is.
Kath Pay:And that's why I'm saying we can be doing the testing, but we can't be treating it like we would normally like a business as usual type test where we can then go and say, oh, that's really interesting. I think we might test it again on this audience and see if it fall or holds through with them as well. Right? Or maybe we've tested the footer 3 times now or 6 times, 8 times. It works a treat.
Kath Pay:Let's go and make that into a permanent feature within our template. So we have to understand that, yeah, or the outside world has so many variables that do do and can and will affect why someone will buy from email.
Scott Cohen:On the concept of gifting too, right? I mean, so you've got people that shop from you that time of year that won't ever shop from you again. You've got, I mean, I have 2 kids Every year, the Christmas list is different. Right? So I might be shopping at different places or Amazon gets to see weirder and weirder things for me to be shopped for, not just coyote urine, and Garin knows that story.
Scott Cohen:And that's a story for another day. But you but I you know, you talked about segments and audiences, like, you need to think about separating out your November, December audiences, your new customers into their own world, right, their own welcomes, their own post purchase, things like that, because they may not be buying for themselves, or they may be buying for themselves for the first time because it was holiday and not any other time of year, and you had a really good deal. Right? You know, theoretically, most companies have deals. That's their biggest time of year.
Scott Cohen:Their biggest offer of the year is Black Friday, Cyber Monday, those times a year. So it's everybody shops that time of year with a different look than Yep. Even the I mean, in the states, we have Presidents' Day, we have Memorial Day, July 4th, and Labor Day. Those are the other big four holidays. Especially, I worked in mattresses.
Scott Cohen:Garin mentioned mattresses. For some reason, people buy mattresses in cars those times a year. So it's and Black Friday is the largest tire sales day at Walmart every year. Serious. Tires?
Scott Cohen:Yes. And it's because they have the day off and the deals are going. Right? So it's it's re you have to, like, you have to adjust for it. I love that you said you can do it for the the sake of, you know, the the results, but not the learnings.
Scott Cohen:And but I loved your call out of this worked really well. Let's test it during normal cadence time. Right? And see if it repeats. You could find some things you test again, but you wouldn't take your winners and go, oh, this one, now this is the new control.
Scott Cohen:Right? Like, you you can't do that. You basically have to throw out November every year in terms of carrying things forward.
Kath Pay:Yeah. And that that I mean, this is for another discussion another day, but that whole gifting that I don't know why retailers haven't learned yet. There are some savvy ones who have gone in and said, is this a gift, you know, at the checkout process? Right?
Scott Cohen:Hello? Yeah.
Kath Pay:This is a gift. Fantastic. Now how much insight is that giving us about that particular purchase and and what can we do with that information? So but they don't do it, and they should.
Scott Cohen:You would think Amazon would know this kind of stuff. Right? Like, my wife maintains an Amazon account to buy me gifts. Everything else she does through my account because I carry Prime. Oh.
Scott Cohen:Right?
Kath Pay:That's it.
Scott Cohen:That's it. Right? Like, so she literally maintains an Amazon account. So the the twice a year she buys me things, my birthday and for Christmas, You know, she she can do it, and I won't know about it. And you would think Amazon should know, oh, this is the alternate gifting account or something.
Garin Hobbs:They're losing out by not recognizing that. 1, you wanna call that action out in your message content. So rather than here are things you might like, here are things he or she will like. Right? That's step 1.
Garin Hobbs:2, all studies show that when people buy are buying gifts for others rather than buying for themselves, they tend to spend around 2x more than if they were buying for themselves. This is a great opportunity for brands and marketers to really understand these standout purchases, frame them, and couch them as buying for others, and watch the average order value increase by doing so.
Kath Pay:Yep. Yeah. Absolutely. And also, frequency. It affects frequency.
Kath Pay:Right? If they're not buying for themselves and they don't have that immediate need, okay, well, let let's just sort of put them aside and and we'll treat them differently. And we will probably not email them as frequently, but around the times that would be holidays.
Scott Cohen:I think that gives us a great last question here. Beyond frequency, what is a lever or 2 you love to pull during holiday season?
Kath Pay:Oh, okay. The frequency, ramping up a particular one. Testing. Again, I would still go with the emotions. Alright?
Kath Pay:The motivations. Particularly knowing that this is the gifting season. Although having said that, a lot of people use this for themselves as well. So it's just a matter of, you know, testing different wording, testing different motivations and, and seeing which ones, work for you, your brand, that particular product. So you might find different products have got different ones.
Kath Pay:Right? But, but, yeah, motivations.
Scott Cohen:Nice. Well, I think that's a great ending point, Kath. Thank you so much. Where can people find out more about you and holistic email marketing?
Kath Pay:Well, you could, you can find me on LinkedIn. So I'm Kath Pay on LinkedIn. Go to holistic email marketing.com, and there's a little pop up there. You can go and sign up and and, get on our list. We have email and more as well.
Kath Pay:We have, some events happening with email and more, of which Scott is, yeah, a regular. You're a regular on it. Ah, yeah. So, Yeah. So that's basically it, I think.
Kath Pay:But I'm I'm everywhere. I'm all new.
Garin Hobbs:Too long.
Kath Pay:You are.
Scott Cohen:And I I will say she's not paying me to say this. This is legit, a fantastic book. Go check it out on Amazon. I have been known to buy them and distribute them to my teams in the past, so it's it's a really good book. And unlike late night hosts who claim they've read a book, I actually have read this book.
Scott Cohen:So, you know, it's it's really good stuff. And thank you so much again for joining us, Kath, and thanks to you, our listeners and watchers, for tuning in. If you'd like to learn more about Inbox Army, check us out inboxarmy.com. Until next time, be safe and be well. Cheers
Kath Pay:all.