Subscribe
Share
Share
Embed
How do product teams decide what to build and what not to? The Experimentation Edge is the podcast where product, growth, and engineering leaders share how A/B testing, feature flags, and experimentation drive real business outcomes — backed by named companies and real numbers. From DoorDash's 12,000 A/B tests a year to Atlassian's experimentation-led product win to UPS's $500M experimentation team, each episode goes deep with operators running experimentation programs at scale.
Hosted by Ashley Stirrup, CMO at GrowthBook and a 25-year executive in data and experimentation. For product managers, engineers, data scientists, and growth leaders at B2B tech companies who care about experimentation culture, statistical rigor, and shipping with confidence. No marketing speak. Just operators explaining what they shipped, what moved the needle, and how experimentation reshaped their teams.
Topics: A/B testing, experimentation, growth experimentation, product experimentation, tech experimentation, feature flags, experimentation culture, statistical significance, marketplace experimentation, conversion rate optimization, experimentation at scale.
Geoffrey Bell & Ashley Stirrup | May 20
===
[00:00:00]
Speaker: when I was at Microsoft, they, um, you know, they had a program that was running at a very, very, very high volume, really lots and lots of tests across Xbox amongst other places.
But that's where I learned about the, um, the experimentation piggy bank, and it was like, hey, you know, over the period of a quarter or two quarters or a year, here's the revenue that we were able to realize from positive experiments. And those... That, that was always good. But here is the revenue that we saved from
Welcome to the Experimentation Edge, where product managers, data scientists, and engineers talk about how they make smarter decisions. I'm Ashley Stirrup, the chief marketing officer for GrowthBook, and in each episode, I'll sit down with an executive to unpack how they use experimentation and A/B testing to make better decisions.
This show is sponsored by GrowthBook, the open source experimentation platform leader. Now let's [00:01:00] jump in and get started with our next guest
Speaker 3: Welcome to today's episode. I'm excited to have Geoffrey Bell, experimentation product specialist at Ford Credit. welcome to the show
Speaker: Thank you for having me on the Experimentation Edge, actually. Uh, I'm, I'm glad that we ran into each other, uh, what was that? Two months ago now down in, uh, St. Simons at the Experimentation Island. So thank you. Thank you for the opportunity
Speaker 3: Well, thank you. Yeah, and I'm excited to have you on the show because you've got a great background. You know, you, you started at, at Lowe's, and then you went to one of the all-time experimentation leaders Microsoft, and then went to someone-- somewhere that was, you know, more new at experimentation.
So I'd love to have you tell us a little bit about your personal journey.
Speaker: Thank you. Uh, yeah, you know, really, really fortunate to have really, uh, kind of found my way into this field of experimentation and AB testing. And even before I was at Lowe's, I was probably a more of a traditional B2B marketer. You know, [00:02:00] started to kinda get creative in our advertising and, and start running multiple ads with a couple different changes, and that's where I started to kinda tiptoe into, into AB testing.
But it wasn't, to your point, until I got to Lowe's, um, which is, you know, a Fortune fifty online retailer, as an AB analyst there, um, you know, we started to learn and really understand the depth that, uh, AB testing can, can take and the mult-multitude of things that you can measure, the things that you can test.
So I spent about three years at Lowe's. That was based in Charlotte, North Carolina. Had an opportunity to, to join Microsoft as an experimentation program manager. That was on a contract basis. Um, and that was specific for the services of Xbox. So believe it or not, for the gamers out there, when you turn on your Xbox, you are immediately flighted into sometimes hundreds of tests that are running across that that ecosystem.
Of course, there's [00:03:00] xbox.com, and they have a mobile app as well. So all of those surfaces were prime real estate for running experiments. And then, uh, ab-about probably just about three years ago, I joined Ford Credit as an experimentation product specialist. And it's been great. You know, at every single stop, I've learned a tremendous amount and ideally have been able to bring those learnings from one organization to the next and really trying to grow the culture and, and the, um, the, the practice of experimentation.
So it's been a great ride.
Speaker 3: Yeah, that's terrific. And so now at, at Ford Credit you, you have kind of a central team and you're supporting maybe 12 different groups. Is that right?
Speaker: That's correct, yeah. We, uh, we internally call, call ourselves the AEV team. We're the analytics, experimentation, and data visualization team. And, you know, through, through tagging, through, um, test intakes, through Contentsquare ideation, uh, we, we really have a lot of fun little bells and whistles that we can pull and press [00:04:00] to help empower our customers, those are-- those being product managers, in discovering features, you know, thinking about their product roadmap through the lens of experiments, through the lens of data analyzation.
And it's been great, right? We, we kinda started early and found some small incremental, I don't wanna call them wins, but some you know, established kinda some foundations of what we can do and how we can help. And from there, it's really kinda taken off, and we've gotten a little bit more mature.
We've, we've kind of picked up our volume and, you know, really trying to, to help the organization make, it sounds very cliché, but data-backed decisions, right? Data-informed decisions. And I think we're starting to really kinda hit our stride a little bit. We've got-- we, we've, we've gone through some tech modernization.
We've kinda done, d-done a little bit of that. So now I feel like, you know, the back half of twenty twenty-six and getting into twenty twenty-seven, I anticipate, you know, [00:05:00] big things for, for Ford Credit and their digital product ecosystems.
Speaker 3: Yeah. One thing we should probably just explain to the listeners is that this is Ford Credit, you know, kind of the sister company to Ford Motor Company, and just talk a little bit about the relationship between those two.
Speaker: Sure. Yeah. That's-- So Ford Credit is, is a captive lender of Ford Motor Company and our parent subsidiary or our, our parent Ford Motor Company based in Dearborn, Michigan. I'll be up there in a couple of weeks. Um, the, the, the goal of Ford Credit is to help customers finance their, their purchase or in some cases their lease of Ford and Lincoln vehicles and, and when customers do that with Ford Credit, it, it creates a kind of a symbiotic advantageous relationship between Ford Motor and Ford Credit.
A lot of times you will, you will see repeat customers. That's certainly an important thing is, is, y-you know, in the lifetime of a, of a [00:06:00] human, I think it's something like you wind up buying five or six cars, and in a l- a lot of times it could be with different lenders every single time. But if you're able to establish p-p-particularly early on, um, a good purchasing journey and, um, a, a, a financing agreement that works for the customer and for the business, the lifetime value of that customer, you know, exponentially increases over time.
So it's a really good symbiotic relationship. Most, most notably, you know, the big three auto manufacturers amongst others, right? General Motors, BMW, Mercedes, they all typically have some sort of captive lending, and so that's what Ford Credit is on behalf of Ford Motor Company.
Speaker 3: Yeah. Yeah, that makes a lot of sense. And, you know, would-- never really thought of it, but once you t- you, you bring it up, it makes a lot of sense that if you can have a great credit experience while you're buying the car, you're likely to be a more sticky customer. So that, that seems like focusing on the customer experience has gotta be incredibly important for [00:07:00] Ford Credit.
Speaker: It is. And, and really, you, you know, we want, we want the process to be as easy and as stress-free as possible. I think unfortunately in this day and age, um, the relationship between a, a prospective pur- car purchaser and the dealership sometimes is not a great one, right? I think sometimes you think of a dealership, you may think of a sleazy salesman.
And I, I'm sure some people have that experience. M-mine i-is not that, right? I've been fortunate to, you know, ha-have, have good experiences in that regard. So it really, it really incentivizes for the digital journey before a customer actually walks into a dealership, to arm them with as much information as possible so that they can have a somewhat seam-seamless purchasing opportunity.
Um, so I think, you, you know, unfortunately, the, the days of, of dealerships are, are-- they're not always viewed as positive. [00:08:00] I think a lot of times they can be. Um, and so we just try and really make that, that journey into that dealership a little bit easier and a little bit stress-free, if at all possible.
Speaker 3: Yeah, it makes a lot of sense. And so when you talk about, you know, supporting, 10 to 12 teams, like give us a sense of like what does a team look like and like what's the range of different experiences that you're assisting?
Speaker: Sure. Yeah. So I think I oftentimes-- I kind of break it into two categories. One category would be what I commonly refer to as, like, our unauthenticated experiences, right? So that's kinda your typical just shopping behavior, visitors to our pages where they're evaluating, you know, what's a lease? What is, what is purchasing a vehicle as it relates to finance?
What are financial terms? What's seventy-two months? What's sixty months? What's an APR? Like, we have quite a few pages and, uh, a couple of different teams that kinda operate in what I call, again, that unauthenticated. So that would be most synonymous with like, you know, just openly [00:09:00] a-available web pages from Google Search.
Maybe they came over from ford.com, right? They were looking at a car, and they, they found their way over into Ford Credit. So though that ecosystem alone probably has probably four product teams that kinda operate in that unauthenticated model. But then in an-- on the inverse of that, we have our authenticated, um, pla- uh, places, and those would be most synonymous with like if you are a customer.
You've purchased your vehicle, you have financed it with Ford Credit, and now you, you come in typically on a month-to-month basis, and you come in and you make your payment. Um, however, things come up, right? You move, uh, you may have a change in job. You may have a cha-change in a banking provider. And so we want to make sure that you, as a customer, are able to come in and update and, um, edit your account so that it's conducive to your, your lifestyle.
Of course, you know, we never wanna see customer [00:10:00] defaulting or not making their payments and things of that nature. So within that authenticated space, we probably have, you know, another three to four product teams where, where, where you come in and you make your payments. We have a recommendation product team that says, "Hey, Ashley, you know, you've been a customer with us for four years now.
You're getting towards the end of your sixty-month agreement with us, and, uh, we love that you bought that F-150, but have you ever thought about a Bronco? Or have you maybe thought about the, um, the newer version of the F-150?" So those are kind of, in my estimation, the, the two kind of playgrounds that we help evangelize experimentation
Speaker 3: Yeah, that makes a lot of sense. And it's funny 'cause you think about credit and you, you think about the buying experience, but you really are touching the customer for their entire lifetime with that vehicle. So pretty important to get that right.
Speaker: Certainly, you know, in some cases when, when customers choose a lease that agreement oftentimes, you know, can be maybe two to three years. And so that, [00:11:00] that sometimes those types of customers, it's a little bit more transactional in nature. However, uh, uh, an, an abundance of, or a majority of customers typically purchase their vehicle, finance it for anywhere from, you know, five, sixty, sixty months up into seventy-two months.
Um, and those, those are equally important customers and also, you know, have opportunity for upsell and for converting them into second-time customers as well.
Speaker 3: Yeah. Ford Credit must be a pretty big business.
Speaker: I would say so. I mean, we, you know, we, we are of course located in, in Dearborn. I think the employee count is in excess of probably 1,000, 1,500 at least. And again, that's just, just credit. We are, um, largely kind of str- across the country, however, located in Dearborn and, You know, the, the Credit Alone has made its own evolution of [00:12:00] actually it was traditionally brick-and-mortar locations throughout the country.
They had a couple in Florida, they had them throughout the country, and I think it was over the last, you know, maybe, maybe 10 to 15 years, everything kind of started to move online, and as, as a result, those branches kind of started to g- to, um, fall by the wayside and kind of get isolated i- in headquarters up in Detroit
Speaker 3: Yeah. Yeah. But the dollars that must go through Ford Credit must be pretty staggering.
Speaker: Most certainly. Um, you know, if you think about your average cost, I think the average cost of a Ford, again, they have so many makes and models, um, but I think it's ballpark $50,000, right? Somewhere in there.
Speaker 3: Yeah.
Speaker: Know, if you sell thousands of cars, if not, if not tens of thousands of cars just in the United States alone, probably more than that, the, the numbers do tend to get up there as it relates to like receivables and revenue.
Mm-hmm.
Speaker 3: Yeah, yeah. So experimentation can play a huge role in you know, the amount of business [00:13:00] that's going through not only Ford Credit, but through the whole b- through the, you know, through Ford Motor as well.
Speaker: Certainly. Absolutely
Speaker 3: Yeah. So tell us about an experiment where you had a lot of learnings
Speaker: So many to choose from. Um, a couple different companies to choose from too, but we'll, we'll certainly stay close to home here with Ford Credit and, um, you know, the one that comes to mind, it was actually, um, it was one of the tests early on when I joined Credit probably, probably about two years ago. Uh, but we have a, we have an online tool.
It's a prequalification application, and largely that's just a way of a customer or prospective customer offers up some, some generic, very simple information about themselves, their location email address, contact info. Largely just kind of a, a, a lead form that helps dealerships, y-you know, kind of route leads to the corresponding dealerships based on their location, their, their mailing address.
And one of the things that at the time, one of our, our product folks wanted [00:14:00] to they actually wanted to ship it, they wanted to just kind of put it out there was, "Hey, let's put, let's put a, what we call a vehicle selector early in the prequalification application." So consider the application just a lead form.
The, the idea was, hey, if we give a customer the choice of a vehicle, then there, there'll be higher likelihood that they will actually convert into a prequalified customer. And, you know, early on I said, "Hey, I think this would make a really good AB test," right? We would run a control that has no vehicle selector in the prequalification flow.
We'd run a treatment that does have that vehicle selector. And over the period of two to three weeks, we quickly realized that the, the conversion rate of the prequal application significantly fell off, right? So the customers were not completing that application as much when presented with the vehicle selector.
Probably worth mentioning is that, you know, you could make the argument that, [00:15:00] okay, that reduction in, in qualified a- prequalified applications, maybe they were in fact stronger leads that were going into the dealership. Uh, however, at the time, we didn't really have that throughput measurement, right?
We've gotten a little bit more mature, and we can kind of, we can kind of get a better understanding of that now compared to a couple years ago. So long story short, we learned, hey, you know what? If we ship this, we're gonna see a, a pretty steep decline in the number of applicants that we're getting.
And so I think that was a bit of a, uh, a shocker to some of the product folks right away that traditionally probably would've just said, "Hey, let's put this out there. We, we think it's a good thing." And through experimentation, we were able to identify that maybe not in fact, uh, quite as a good thing as we originally thought.
Speaker 3: Yeah. And you were telling me that, you know, 'cause experimentation was pretty new at Ford Credit, that was a pretty important learning that you were able to bring to the organization and kind of opened up their eyes that, wow, you know, [00:16:00] experimentation can really help us understand better, know, how we're impacting the customer experience.
Speaker: Isn't it, uh, isn't it always the case y- that, um, it's the losing tests that wind up you know, creating or finding the most value, the most insight? And I think this one was no different. And, you know, as a result, I, I always try and advocate for like, "Hey, let's find another, let's find another way to run this test.
Let's find another place for this feature." Um, I think we, you know, shortly thereafter tried that vehicle selector after the pre, the prequalification application had been submitted, and we saw some nominal some gains in that regard right around, okay, now I know I'm prequalified, and this is in fact the vehicle that I'm interested in.
There's a couple, um, variations of that test that we've run. But that was early on. I think that, that learning, that loss, if you will, um, it instilled some confidence and some some visibility, some increased [00:17:00] visibility into our program, um, that, that leadership saw and said, "Okay, hey, you know, let's, let's continue to where and when we can, let's go test features.
Let's try and let's try and test them a little bit earlier, too, in the development process." I think you and I will probably talk about the notion of, of, of painted door tests and things like that. So in a strange way, you know, you get a little more attention. I don't know if that's the right word, attention, but you get a little bit more visibility when you have-- when you provide truth in data.
That's just really the name of the game. That's one of the things I like about experimentation. One of my former directors was like, "Hey, listen, it's-- a lot of times there can be some gray area." I'm not gonna s-- I'm not gonna deny that. But a lot of times it, it tends to be black and white, and we are just providing data, uh, that tells the story one way or the other.
Speaker 3: Yeah. Well, sure, the biggest learnings always come from losers and then what you
Speaker: Yeah
Speaker 3: those losers. I mean, you know, sometimes you might run a test and say, "Okay, that's, [00:18:00] enough. We've learned enough on this one." But in other cases you say, "Okay, I, I ha- I have conviction in this area. Now how do we iterate?"
And you gave a great example where you, you move that to later on in the process. And in particular, what I love about that story is it's, it's really helping you to learn how to align the experience with what the customer needs, right? Like when they're doing a, a credit application, they wanna know where they are in the credit process, and you start to say, "Hey, have you thought about this vehicle or that
Speaker: Yeah. Yeah
Speaker 3: send them down another path. And so really understanding what's the customer doing in the moment, and then how do we use experimentation to learn about the best ways to support them. Super powerful.
Speaker: It kind of reminds me actually, uh, in a s- in a s- some roundabout way, we had a saying, a joke when, when I got started at Lowe's, which is largely e-com, right? So lots of add to cart, lots of revenue per visitor, uh, traditional e-com measurement. We had a [00:19:00] joke that was like, "Hey, can we just put an add to cart button there?
And, um, hopefully orders will pick up. Hopefully RPV increases." And, you know, it was rarely the case that just putting an add to cart button earlier in the customer journey, it, it seemingly never led to more orders or more add to carts. And so the, the same kinda concept here is like, okay, where is a customer in their buying journey?
Is, is it too soon to put a vehicle in their face, right? Or, uh, uh, you know, let them choose their vehicle. So that, that was a good learner for sure.
Speaker 3: Yeah. And more recently you've been kind of experimenting with personalization, is that right?
Speaker: Definitely. You know, personalization I think is one of those things that has just massive, massive opportunity and, You know, v- I think probably very few companies are, are doing it well, right? You think about the Netflixes, the Spotifys, the Booking, like, yeah, they, they probably have [00:20:00] unlocked it.
Microsoft, they, they've, they've figured it out how to s- how to, uh, scale it. Uh, but just kind of getting going here early on, I alluded to it a little bit earlier, "Hey, you know, we think that you're a good fit for this type of vehicle given that you either have had this purchased from us previously or, you know, you purchased a minivan.
Maybe now it's time for a larger van. Maybe your family's growing." So we have, we have some of those things that we can tap into. But what to me has been most fascinating is, is Ford Motor Company has they have some personalization services available for consumption by their by their subsidiary organizations.
Ford Credit is, is no- is, is one of those, where we can say, "Okay, we know a little bit about a prospective customer," right? We know a little bit about the cu- that prospective customer was looking at F-150s, and maybe they were looking at F-150s that were blue in color, and maybe that, that powertrain engine.
And so you can start to [00:21:00] create experiences through, whether those be A/B tests or, or testing different personalization campaigns and messaging. And the couple that we've run, we most certainly have seen very large lifts in engagement, right? Whether they be clicks or page views or time on the page or, or, or re- reductions in bounce rates.
Um, and so I'm encouraged by that. We're definitely tapping in and stepping in and trying to kind of get a larger and larger, larger audience to experiment on, call it a sample size, if you will. Because the challenge for me, and I think for, for, for this concept, is when you want to, you know, send a message to someone who likes their Ford Explorer, they like four-wheel drive, black, automatic, with a certain audio, uh, package, that audience gets smaller and smaller and smaller, right?
And it's, it's harder and harder to [00:22:00] run experiments to that audience alone and try and come up with you know, a legitimate sample size that you can kind of believe in these, in, in this thirty, 40, 50, 70% lift in engagement. So that, I think, is what we're trying to kind of unpack now is like, well, maybe how do we, um, package this maybe with another vehicle or a series of vehicles that says, "Hey, if you meet the criteria for any of these vehicles, let's run, uh, some personalization against that and see how it performs against a control," right?
Where there is no personalization. You may just get static banners or static messaging. So that's been incredibly fun and challenging to, to get into. But there's, to me, there's, there's just, there's just so much opportunity there that I, I intend to, you know, continue to figure it out
Speaker 3: Yeah. I love that. I know that when I'm car shopping, I tend to fall in love with a certain model, a certain color, you know, certain color seats, you know, just the whole thing. And
Speaker 2: [00:23:00] Yeah
Speaker 3: the more you're, you're-- If you were putting that in front of me more, the more I'd be leaning in. So that, that, that's a fun area to test.
Speaker: That's kind of the, that's kind of the, the thing. You know, I think honestly, like some of the larger social media companies, Meta, they, they, they have figured it out. They know kinda like, you know, what y- what your propensity is to purchase something, whether you looked at an ad, maybe you w- you had more than one impression, maybe you clicked, things like that.
They've, they've definitely probably have a sca- a more scalable personalization program than we do. But we're, we're starting and we're, we're figuring it out. We're using the software that we have. We're using the, the services that Ford Motor offers us and, um, you know, the sky is really the limit there.
Gonna continue to kind of press into that.
Speaker 3: Yeah. And you know, I, I believe that you're continuing to add more teams that you're supporting and kind of introducing experimentation to more people. And of course, nobody ever wants to have their [00:24:00] baby be a loser. How do you, you share with them, you know, the results of an experiment when maybe it's ambiguous or maybe it's down? and like how do you help them, kind of take some positive out of that?
Speaker: Good question. So I think I think packaging, you know, packaging your test result, um, is pretty critical. Generally speaking, I think human beings, their attention, their, their attention, their time of attention is somewhat small and limited. And, and if you think about a thirty-minute meeting or maybe even an hour-long meeting, like that first five or ten minutes is really gonna set the tone for the, the remainder of that meeting.
And so I kinda take that same concept into a readout, right? And we kinda repeat, "Hey, here's what we tested. Here's what control looked like. Here's what treatment looked like. Here are the results," right? We saw this metric go up. We saw this metric go down. What does that mean? I like to try and tell the story as to the [00:25:00] best of my knowledge, to the best of my ability, given the data, given the analytics that we have to play with and offer perhaps some insight as to, well, why did these metrics move given the change that we put in front of that customer?
And sometimes the story tells itself. Sometimes it's very plain to see, oh, well, we moved something up higher on the page, and so as a result, something lower on the page got less engagement with. Or you know, we reordered the, the steps of, of this application or this process, and as a result, we saw fall off or drop off in this portion of, of the flow.
Uh, but to your point, sometimes it's a little more-- there's a little more gray area, and there's a little more ambiguity. And that, to me, is where opportunity presents itself to run a test again or iterate on the design or offer contextual opportunities to try and learn why did [00:26:00] these metrics move the way that we did.
We didn't maybe expect this. So you know, the-- certain, certain applications like Content Square w-we-- that we use is, is very helpful. But, uh, most, most product managers, I think, know their product pretty doggone well and, and can help shape hypotheses based on previous tests, uh, that have run. But, you know, to kinda come back to it, how do you, how do you tell the story?
I think you have to get creative, and I think you have to get generate some level of excitement early in the process or as you get down into the weeds around, you know, metrics. I think the attention, um, starts to kind of fall off to, you know. So that, that to me has been The, the best way I can convey what happened is to try and generate some level of excitement and tell a story rather than just run through the numbers, two per- 2% here, 5% there, here's a lift, here's confidence levels.
That, that to me is kind of the dry [00:27:00] scientific backbone of experimentation. But if we can, if we can make science fun, you know, I think that's where we get some interest and, uh, more opportunities, more swings.
Speaker 3: Yeah. Yeah. Well, it makes a lot of sense. I mean, basically through storytelling, you're helping people to really learn, like, what is the real customer journey and what is this data telling me about that customer journey? And so that, that seems really powerful, particularly if you can get them excited about some iteration that can then lead to a win where you take, you know, a loser and turn it into a winner.
That's, that's so powerful.
Speaker: You know, um, the, the other thing there too, I think that I, I've been able to have some success with is when we think about like the development, like who's gonna develop this experiment, right? Or okay, right, we just ran a test or we ran a series of tests, and now we have a couple of, of opportunities that we think that we're gonna be able to tap into, but okay, hey, sprint planning is coming up, quarterly planning is coming up.
Do I as a product [00:28:00] manager, do I need to plan for this in a forthcoming sprint or iteration or quarter? Um, and so we've got a little bit of a blended approach here where, where like my development team can help create and develop those experiments. It, it kind of depends on a, on a, on a case-by-case basis.
But also good tooling, right? Like GrowthBook or the, you know, others in, in the s- in the space. Optimizely comes to mind. We are, of course, in the house of Adobe, so we're kind of exclusively using Adobe Target. Uh, but when you, you have friendly, user-friendly testing s- testing tools, you can enable product managers to do some of that on their own.
Maybe, maybe they can tap one of their devs and say, "Okay, hey, this is a, this is a one-pointer, this is a two-pointer. Come in here, use this visual experience composer, use this drag and drop editor to move some stuff around the page." And that doesn't necessarily require hard coding, right? Maybe there's some JavaScript that you can apply to a page to move some things around.
So I [00:29:00] think that also has helped us, uh, not nec- not only increase our volume, but increase the, the likelihood that our product teams are gonna, gonna continue to experiment, is if we make it easier for them to develop it, whether it's them developing it, us developing it. Sometimes there's a bit of a you know, a 50/50 in that regard.
And, you know, good tools like GrowthBook and others, I think will continue to help make the technical aspects of testing development less technical and more easily accessible to, to non-developer personnel. ~That was a mouthful. Sorry.~
Speaker 3: Yeah, yeah. Well, no, but, you know, the-- I really think that's the future, especially as you start to do more and more with AI and AI coding, it's really opening up whole world to being able to do things they never could
Speaker: Yeah. I keep hearing the vibe coding, you know, and I, I know that there's, I know that there's value there. You know, I've toyed around with it a little bit, but, uh, I still think, you know, early, early in the process there. I [00:30:00] mean, AI is obviously everywhere and it's applicable to so many different businesses and so many different areas.
And so, um, you know, we're, we're definitely looking at that and, and leaning into that as well.
Speaker 3: Yeah. And, you know, to, to wrap up, maybe we talk a little bit about the future of Ford Credit. And I think one thing that's really interesting about your business is there's much that happens online, but some really important parts are happening offline, and you're, you're working on trying to connect at least the data from that offline experience to the things you're doing online.
Could you tell us a little about what, what you're working on there?
Speaker: Yeah. Thank you, Ashley. You know, you know, coming from traditional e-comm, fulfillment and transactions take, take place online, right? You're-- You use Apple Pay, which I love. I just love Apple Pay. It makes, makes things so easy. But you-- To my knowledge, there's very few automotive makers out there, you know, the Teslas, the Rivians of the world that really enable you to [00:31:00] purchase a vehicle online.
Carvana, you know, there's gonna be some that, out there that, that can, right? They may not be manufacturers. They could be more in the distribution of vehicles. Um, but one of the things that we've really been trying to, and we are, we are, we are there. It took, it took quite a bit, uh, of, of an effort, but what we're, what we're trying to do now is You know, demonstrate and visualize the impact on revenue and in, in the case of credit, where we, we, we've commonly referred to it as receivables, right?
How does an experiment online impact the rece-- the, the downstream receivables that we anticipate when a customer goes and transacts in their dealership, in their, their location? I, I ca- I don't know the number. I think it might be in excess of 5,000 dealerships across the country. Right? But when we provide a, an online exper-experience, and whether there's a test [00:32:00] involved or not, we wanna know, does that ultimately lend itself to more customers purchasing vehicles and/or purchasing those vehicles and financing it with credit?
And so I think for a long time, we didn't have that throughput. We didn't have that ability to kinda con-connect that offline behavior with the online digital experience that we were providing customers that they would commonly use before walking into a dealership. So that's been really exciting internally for us to be able to tell product teams, "Hey, you know, when, when you run experiments," particularly if you're in that purchasing flow or maybe a renewal area, it's, it's a little bit harder to do, right, if we're out on a unauthenticated page and you're just simply kind of perusing and, and looking for content.
But when you get into like a buying flow or a renewal flow, if we [00:33:00] can present an experiment and say, "Hey, you know, control saw a thirty percent close rate, right, for customers who purchased and financed with credit. But treatment saw a thirty-two percent close rate that purchased and financed with credit."
That two percent incremental revenue gain, when you kind of project it out and annualize it, it has a, it, it has a large impact. And that, that has, to me, has been getting our product managers very excited. Now- I think that revenue, i-it shouldn't be the end-all be-all of an experimentation program.
It can definitely be very powerful and insightful, but there are plenty of other areas where engagement and, you know, page views and click rates and bounce rates and scroll depth and all those things are incredibly important to a, a particular page or, or a p- or a set of pages. Um, but, you know, revenue is, [00:34:00] is a, a very important business metric, and experimentation is one of those unique places that marries customer behavior with business metrics and, like, how do those intersect?
That, that to me, I think, is where experimentation is, and, uh, that's why it's so fun and frustrating sometimes and complex and technical. But really it's a great relationship to be able to tell stories about, you know, how changes in customer behavior ultimately impact business metrics, whether they be receivables and revenues or just clicks and, and page views and things like that.
So very exciting times here at Ford Credit. You know, and I'm excited and thrilled to be, be kind of setting a new, a, a new level of measurement that previously I don't, I don't think that we had a-- we did a particularly good job about
Speaker 3: Yeah. Well I think it'd be super interesting to have you back on in a year to talk about that journey 'cause it takes longer, [00:35:00] it's offline, and so you don't know all the other experiences they had in between your experience and that experience.
Speaker: Yeah
Speaker 3: be a ton of learnings that go into that, but I think the opportunity's massive for your business
Speaker: I look forward to it, Ashley. Um, when I was at Microsoft, they, um, you know, they had a program that was running at a very, very, very high volume, really lots and lots of tests across Xbox amongst other places.
But that's where I learned about the, um, the experimentation piggy bank, and it was like, hey, you know, over the period of a quarter or two quarters or a year, here's the revenue that we were able to realize from positive experiments. And those... That, that was always good. But here is the revenue that we saved from shipping poor experiences.
And, um, that is where I'm, I'm really trying to get our, our program to a place to where we can kind of start to demonstrate the return on investment for the business for what we do here and how we [00:36:00] work with, with other product teams
Speaker 3: Yeah. I, I'm excited for your journey. I think the potential to continue having a huge impact for Ford Credit is just massive. So
Speaker: I appreciate that
Speaker 3: yeah, so really appreciate having you on the show. I think we, we learned a, a lot today about a pretty interesting business. So, so thank you so much.
Speaker: Thank you, Ashley. All the best with Experimentation Edge, and I think Growth GrowthBook is a wonderful platform, and I look forward to seeing you guys out on the the conference trail and at some of the other events. I look forward to meeting you again
Speaker 3: Sounds great
Thanks for tuning in. If you enjoyed this conversation, please support our channel by hitting the like button and subscribing. Better yet, share the episode with a friend. I'm Ashley Stirrup with Growth Book. We'll see you next time on the Experimentation Edge.