[00:00:00] Phil: you're convinced that synthetic users, we'll become a standard way teams ideas and explore new markets. [00:00:06] John: I think if we were the branding team, we probably would've come up with something else. So how about dynamic personas or something that accurately represents your audience, and tries to simulate it. [00:00:15] And another way to think about though is, you know, hey, I've got this body of evidence maybe in my company, could there any way I can harness that to think about new novel possibilities? [00:00:24] when we do research is this a a viable strategy? Should we go in this direction? Those kind of things naturally people want a classic, you know, Hey, give me the deck of the results, but it's static, right? [00:00:34] So, so if I say, Hey, can I represent this in some sort of, um, synthetic user? oh, I've got another question. What about this? What about this, So we actually, create a tool so that they're able to keep asking those questions, right? [00:00:46] ​ [00:01:13] In This Episode --- [00:01:13] Phil: What's up everyone? Today we have the pleasure of sitting down with Dr. John Whalen, ai customer research expert, educator and author. John is a cognitive scientist. He leads brilliant experience and he authored the book design for how people Think. He also teaches two courses on Maven. [00:01:30] AI for customer research and AgTech AI for research. In this episode, we dive deep into synthetic users, like how to build them from real interview data, why they make stakeholders hungry for real research, how they killed the average persona, how they enable statistically relevant qualitative data. And we'll also cover how to design marketing content for AI agents and human users simultaneously. [00:01:55] All that, and a bunch more stuff after a quick word from two of our awesome partners. [00:01:58] ​ [00:04:02] Phil: John, thank you so much for your time today, sir. Really excited to chat. [00:04:06] John: Yeah. Well, I've had my coffee. I think I'm ready to go. [00:04:09] Phil: Awesome. So, yeah, I've, uh, read your book, John, ahead of this chat, and I know that you spent a ton of time exploring recently where AI works in market research, customer research, and where the human kind of fits in, in that whole new process. Uh, and has led you to a pretty counterintuitive conclusion, uh, which is gonna be kinda the focus of a, the start of the conversation today at least. [00:04:31] 1. What Are Synthetic Users and Why Do They Matter? --- [00:04:31] Phil: So you're convinced that synthetic users, and we'll unpack what that means exactly, but like non-human AI users, we'll become a standard way teams ideas and explore new markets. They allow teams to pressure test concepts faster, uh, and more frequently than in a lot of different, traditional studies that folks can use. [00:04:50] And we can't reach that many people that we can with synthetic research. Maybe we can start there. Like, what, what the heck does that mean, John? Like synthetic user research? Uh, can you unpack that for us?[00:05:00] [00:05:00] John: Of course. Um, so first of all, it sounds creepy. You know, we, we had this thing called with synthetic users. I think if we were the branding team, we probably would've come up with something else. So how about dynamic personas or something that accurately represents your audience, um, and tries to simulate it. [00:05:14] So, um, I guess where I'm going with this is, um, just looking to the future. We know what, you know, Chatt PT three was like, I don't know, eight, two years ago, and we, you know, where is it now and, and what could be in the next, you know, six to 18 months, um, in terms, and so right now we're finding that when we use, we actually in, in a class I do every time we collect real human data and then we get, um, uh, a couple specialized tools to actually represent what the answers would be from without the benefit of that real human data. [00:05:48] And we're finding, you know, it gets like 85 to a hundred percent right, of the like major topics and needs in this case of consumers, but it could be B2B. And so. Um, just even with that, it's not a scientific study, [00:06:00] but it's like, Hey, we ought to take this seriously and really consider it and it's worth our. [00:06:04] Our thought. And so I guess I keep thinking, okay, if these keep, are only gonna keep getting better and better, which they are, then logically, um, what kinds of decisions right now go completely by gut and, and no, no, you know, research and what could we use to, to help us, um, frame that. So I guess I, I'm thinking that increasingly designers, marketers, um, engineers will get the hint and be like, Hey, why don't I at least ask this as a directionality or an interesting hypothesis or a way to explore it. [00:06:34] 'cause I know I'm not the user. So any of those things I think can be really valuable. And so we're finding increasingly that very large companies are super interested because I can only, if I, if I've tested things in the US and Europe now and I really should be doing it in, um, Thailand and Brazilian, Portuguese and, and so on, um, 'cause it's really global, maybe this is an opportunity actually to be in some ways more inclusive if I can represent [00:07:00] them at least directionality and directionally and maybe more than that. [00:07:04] Darrell: Do you, do you think that it hasn't, you know, caught on as much because there's either a reluctance to use these synthetic users for research, or do you think it's really just an awareness thing, like they don't know to do [00:07:19] John: Yeah, so just think about it this way. So, so first of all, what's the, I, I think there are a lot of things that you can think about that could make this, uh, so by the way, I'm I'm a curious skeptic, right? I'm, I'm a psychologist. I want to know what's really right, and I don't want to just leap to, oh, here's a cool new tool, [00:07:36] but thinking about it logically, it's something that could be accessible 24 hours for you. [00:07:41] Um, it's relatively low cost. It's low risk. I can try, Hey, what about this concept for a pitch? And just see what comes up, right? So, and it's not in the public. I haven't tried anything with, uh, some sort of landing page or something. So I guess it, it seems very accessible and available. And, and so I would say, [00:08:00] um, Darrell, it's, it's just so new, right? [00:08:02] So do I trust these things or these creepy. Synthetic users. And another way to think about though is, you know, hey, I've got this body of evidence maybe in my company, could there any way I can harness that to think about new novel possibilities? So, so originally there was real concern about, and rationally about bias with all these models and gradually they're getting not just like a Silicon Valley person, but like the world. [00:08:27] And so, um, could this be a way to get us to get an indication of what these more, um, all these personas and possibilities around the world might be like to give us a more accurate picture than we really can today? [00:08:42] Phil: I like how you said if the, the, [00:08:43] the branding team was involved in like, how do we name this [00:08:46] Thing. We probably wouldn't have named it synthetic users because in, [00:08:49] In the marketing analytics world, synthetic traffic has a negative connotation. It's bot traffic, and we're actively trying to suppress those people from our actual numbers, [00:09:00] right? [00:09:00] So like synthetic users right away. I think that, you know, maybe some folks that are deeper in like the, the research side. I, I've played a product marketing lifecycle role in, in my in-house career, and we had a lot of debates about qualitative user research versus quantitative and going back and forth there on that. [00:09:16] And I, I think that there's some really interesting applications for this, but. I there's also, like you said, the skeptical angle to this. Like some people are thinking, all right, we're researching human subjects, but we're replacing those human subjects with robots. And on paper that kind of makes the whole thing less human, right? [00:09:36] But you've actually argued that synthetic user makes this whole thing, the market research out of it, more human centered. And it's almost like an oxymoron, right? Like non-human users can humanize market research. Um, but your argument, and you've shared this, uh, ahead of our chat today, synthetic users make senior stakeholders more interested in real human research. [00:09:57] And, um, I'm, I'm, I would love for you to just unpack that [00:10:00] in your experience. Like [00:10:00] 2. How Synthetic Users Make Stakeholders Hungry for Real Human Research --- [00:10:00] Phil: what happens when a stakeholder starts to [00:10:03] interact [00:10:04] with that, like live persona or synthetic user, like an AI representation of a customer, I think you called it. And, and why do leaders like, become more eager to invest in, in, real human, um, uh, research because of it? [00:10:17] John: Yeah, I mean, this is anecdotal with a half dozen, you know, kind of clients. But, but the idea here is that, um, so first of all, uh, when we do research to, you know, should, is this a a viable strategy? Should we go in this direction? Those kind of things that are sort of major impactful things. Um, naturally people want a classic, you know, Hey, give me the deck of the results, but it's static, right? [00:10:38] So, so if I say, Hey, can I represent this in some sort of, um, synthetic user? What, so what does that sticker wanna do? They want me to go away. And then, oh, I've got another question. What about this? What about this, what about this? So we actually, um, create a tool so that they're able to keep asking those questions, right? [00:10:56] And so logically, and we do frame it in a way that I [00:11:00] think is like safe with guardrails, that we say, Hey, this is directional. This isn't data. This is like a interesting possibility, you know, hypothesis you. Creation. Take it as you will, uh, very gently, you know, in the way we'd say it. But, um, inevitably they start being like, Hey, would you like this? [00:11:17] Why would you like that? What about this, what about this? And then at a certain point they're like, wow, if that could be true, this could be a really interesting angle. We need to go test that with our real humans. So think of this as almost a preview of what you could have with your, your humans. So you're being more prepared for what might be to come, what might be the distribution of different responses. [00:11:39] So, so a chat, GBT, or, and I'm not faulting them in particular, but classic LLMs tend to give you a pretty similar answer over and over. So the more prof, the more sophisticated tools that are on the market right now, try to give you a more realistic human distribution of responses. So it starts to get more interesting at scale. [00:11:57] And then you start to get to like cultural variation and things like that. [00:12:00] So if you can layer these pieces on top of each other, it could be a more. Interesting things. So I guess the moral story is by letting people try these things, you're like, is that really true? So it's really not. I've come to a conclusion. [00:12:13] I'm never testing a real human again. It's that, it, it's, it's preparing me for, for being better informed, um, and to understand the, the variants that I might get actually with real humans. [00:12:24] So I'm not trusting my gut anymore. [00:12:26] Phil: So it's like a stepping stone, almost like a proof of concept to convince those stakeholders that maybe don't always want to spend budget on those big, uh, research studies to just like, dig a bit deeper and, and then like curious about it. Um, yeah, I, I think that the, that's a cool way to frame that there. [00:12:42] Like, um, not saying that we shouldn't still be doing like some of those long-term studies, [00:12:47] um, but I guess it's a cool way for using AI to build that internal political capital, if you will, to fund that, that deeper research and, and, and longer term experiments, like I've been in that hat and in-house [00:13:00] careers where, [00:13:00] um, you know, we wanna run like a, a new experiments, like a holdout test and it's gonna cost a certain amount of money 'cause like we're holding out a percentage of the audience. [00:13:08] But like, [00:13:09] you know, getting that buy-in to do it, I just can't like, go off and do it on my own. But, you know, I'm trying to like, find an application for, is, is that the right way to think about it? [00:13:18] John: Excuse me. Exactly. Um, so I think that, um. Uh, having, having this possibility of, you know, so, so what are the market, for example, we never get to test all the groups we want to, so what are the, what are the segments that maybe we can't get to, but we might get an indication of? And then that might be a good way to say, oh, actually, of these three, there's this one actually looked more interesting than we thought. [00:13:42] So let's, let's divert a little bit of our resources there. Um, you know, another very different way to think about this is they're just a good tool, right? So it is just a tool. And so when we first meet with a stakeholder and they're like, Hey, let's, do, you know, I'd love to learn more about this with this audience. [00:13:59] Um, [00:14:00] well, what we do is say, great, let's spin up a, a quick synthetic user right now and ask them that [00:14:05] question. And it's not because like slam dunk, five fives were all done, but rather almost always that stakeholder system like, oh, oh, that. That isn't quite what I want. I, I would, I wasn't really thinking about that question. [00:14:18] I was really thinking about this question, and so I don't know if that's really the right audience. And so suddenly I've just learned so much more about the stakeholders interests, right? So it doesn't matter almost what that synthetic user said. It's more a tool to get us at really understanding the heart of the problem that we're after solving. [00:14:35] And so, in any way along the line, if we're gonna do qualitative interviews, we can test it out with synthetic users. We can ask. What I call sort of impossible questions. What would you never tell an interviewer or never put on a survey, but we should know [00:14:50] what's, um, uh, you know, what are some of the underlying possibilities of why you're thinking this way? [00:14:56] Um, for the people that are, are of the inclination [00:15:00] of jobs to be done, um, give me your answer and then describe it in jobs to be done format. Right. This is an LLMI can start saying it that way. Or if you are a product person, give it to me. And, you know, as a, as a blank, I have to do this so that I can. And so I'm just saying that, you know, there are very unnatural things we can do with these things as well to get really to the sort of business need that we have. [00:15:22] So it's just, just a different way of thinking, different tool. [00:15:25] Darrell: I especially like how, you know, testing with these synthetic users helps us pressure test our own thinking ideas or frameworks because, you know, just like you said, we might not have thought about this dimension or this lens and it, it might even change like our whole business idea. So, so just the act of. [00:15:49] Asking these synthetic users is, is useful, like the exercise is useful. Um, so that's super interesting. I wanted to ask you, [00:15:56] 3. Pre-Testing on Synthetic Users: Shortcut or Smart Step? --- [00:15:56] Darrell: how do you answer the charge that pre-testing on synthetic users is a shortcut that risks missing? The visceral emotional hugs and tears you famously drawn out in real world contextual inquiries. [00:16:09] John: Yeah. Um, well, certainly, you know, there are a few things that we know LLMs today, in, in early 2026 are, are not great at, right? So, um, they don't have the grounding, right? So if we say, Hey, I've got, you know, 20 foot ceilings, um, should I, you know, put my cat up there, you know, how would I do that? Well, we'll just reach up. [00:16:28] And so first of all, bad idea to put your cat up there. Secondly, I can't reach 20 feet, right? So, so there are these things about just. Basic common sense that you and I all possess about the world that these things don't have, right? So it's a statistical property, so, so we do wanna be really careful about the weaknesses of these things today as well. [00:16:48] Um, the other is that, um, there may be, um, really sort of counterintuitive findings about real humans in the way that we think, um, you know, we, we are all subject to our [00:17:00] frailties and imperfections, and often it may fall into the trap of being too logical or too systematic. And so. So we want, that's exactly why we wanna say this is a possibility or a notional. [00:17:12] Um, you know, the, the other thing Darrell, I'd say is that I can turn this around too and create synthetic experts, right? So we actually find it very useful if we've got a big stakeholder we're about to present to, we create a synthetic user of, of that or synthetic expert, like that person say, here's what we're about to present as the major points. [00:17:31] What do you expect would be their first rebuttals or questions or concerns? What, what, what am I not answering for them? And you would be shocked at how many times we get a set of questions to be prepared for and think about, and that person actually utters one of them. So, so actually it's a really good way to think about your your peers kind of thing, you know? [00:17:52] Or for example, I don't, I don't profess, I'm really more on the cognitive science side of the world. I'm not a, a brand strategist by trade, [00:18:00] right? So I love to hear what's that perspective of this data, what's another perspective that I might not. You know, my bias is to think about product innovation, not like maybe systems thinking with the, the technologies. [00:18:13] So, so there could be ways to, um, overcome some of our, you know, think more broadly from that perspective too, as having sort of co thinkers for us. [00:18:23] Phil: Super cool John. So like, you're not just talking about this from like a theoretical, like this could be potential and you don't even have like someone on your team do this work for you. You've gotten your hands dirty so much so that [00:18:36] you're actually [00:18:37] teaching a course on Maven about doing this. Um, we actually had a listener reach out and encourage us to invite you on the show because they loved your course so much. [00:18:46] Um, so what does [00:18:47] this actually look [00:18:48] Like Like maybe give us a teaser of, of that like second cohort that you're doing on Maven. [00:18:53] 4. How to Actually Build a Synthetic User: Tools, Layers, and Agentic Systems --- [00:18:53] Phil: Like how does someone actually build a synthetic user? Like, are we getting started with, um, you know, is this like a, an innate n tool? Are using cloud code? Like is this just for technical people? [00:19:05] Like, walk us through that. [00:19:07] John: Yeah. So first of all, there are a set of, of sort of, um, SaaS products that are out there. So there's, um, for you all in, in marketing and really global scale. There are things like, um, uh, um, there's one called Verve Video Survey. There's another called ybl, another subconscious AI Ask, um, rally. There's a whole series of these. [00:19:27] Um, there, there's another that's really good at trying to make, uh, synthetics out of your, um, HubSpot, for example. [00:19:34] Um, and so Delve. And so each of these has sort of a different angle, but we've been, one of the things I've been doing is actually interviewing the founders of those tools and testing them out, um, of course in preparation for interviewing them and trying to understand their angle and how, how they're trying to do it. [00:19:50] So we would just try to be. [00:19:51] Aware and not, you know, logically you can be overwhelmed by all these possibilities and terrified, or we can try to take, you know, a little bit more [00:20:00] control and be the orchestrator of these [00:20:01] things. Okay. So that's one side is the sort of professional tools. The other, and that's where, uh, my first class, we really do bring in like the very latest, best tools and let people try them and, and not say, Hey, you've gotta learn this and like, it rather, let's let you really use them and understand what they're good and not good at from your perspective and how you might use them or not, or know how to tell people we shouldn't be using them. [00:20:25] So, so I think it's, it's where I just want an evidence driven decision. Often I'm good at slight tangents, so in LinkedIn, anytime I post anything about synthetic users, that's when I get the most responses and like really visceral, like, this is crazy or this is great, you know, and I think it suggests there's some, there's a hot button there for all of us, you know, if we think about why are we really so viscerally. [00:20:47] Tweaked by this, and it may really be, 'cause our jobs could be very different. It could be that it's, it's getting at the heart of, you know, so I think it's, it's, it's just a self-reflection moment for everyone. I'll let you answer it for yourselves, but [00:21:00] in one hand we've got things that are these real SaaS tools. [00:21:02] On the other case, we've got, we try to build these things from scratch. And so that's where you can start, for example, from a set of interviews or market research data. So I could just do this in an LLM with a single prompt, right? So that's, that's layer one, right? Um, layer two is I can have, um, uh, there's a lot we know about psychology and so a lot of the professional tools, there's sort of first layer beyond that. [00:21:25] Molten core is all about, um, what are the classic behavioral responses. Like we have more strong reaction to negativity than positivity, you know? Um, here are some of our biases. So let's have that layer be put on there. And then the next layer is logically these massive groups, um, have, have just enormous amounts of user data that's sitting on a shelf somewhere or in a server somewhere. [00:21:48] And let's put that to use and say, here's another layer of what we know about our target audiences. So if you have those set, set of layers, suddenly that could actually be more promising than just outta the [00:22:00] box, whatever, with a, you know, one line prompt. [00:22:03] So, um, so I encourage everyone actually to try, um, I, I do a little free lessons and, and one of the things I do is I, I say, you know, give me a five, um, you know, question, um, sort of questionnaire, LLM, uh, to help you understand the target audience I want and what I wanna look at. [00:22:19] And then, and then I want you to assume that persona. And so it's a really good exercise for stakeholders too, even to just say, okay. Let's do someone just like you, great, um, you know, marketer and blah, blah, blah. Now let's do someone from Brooklyn who's, um, you know, a Jan Alpha really into brands and ask them if, you know, Supreme is still a w or not. [00:22:39] And so, so, and you, and the response you get in the language that you got and everything else just makes you realize you are not the target audience. And so, so, and that there's something that's, that's there underlyingly. So, so what we do is we actually do use, um, things like, um, cloud code or, um, Gemini or, um, codex is the, um, the [00:23:00] other version. [00:23:00] And I actually don't use these things to actually do coding proper, but we build agentic systems to work through the steps of, um, first here's our interviews and what's the interesting nuggets about the personalities, motivations, types of words they use, um, syntax, um, um, you know, things they're trying to solve, things about their life. [00:23:21] They mentioned. And then from there try to encapsulate, okay, so here's, uh, maybe three or four major archetypes in that body of evidence. Let's try to build those out and bring them to life. And then from there you can say, Hey, I'd like to interview them one by one, or I'd like to have them be a focus group and talk to each other. [00:23:39] And essentially you can build up something there that feels very actually, um, realistic. And so a lot of the people in my classes are like, that really sounds like what I would get in my. In, you know, in my focus group or in my interviews, like, it's, it's kind of creepy. And so the right, because there's so much data out there that's actually getting more and more powerful. [00:23:59] [00:24:00] So, so again, and, and to Darrell's point, let's be really careful. It can sound really good, but is it really good? so so we want to make sure that we're actually hitting the mark from a business perspective. Is this actually accurate for truly, would I have the right go, no decision on a marketing campaign, or would I have the right distribution of kinds of people? [00:24:20] Um, and, and the sort of segments that I would find in real life. So these are things we don't have well tested yet that, that, um, are begging for an answer. So I'm, I'm, but I'm trying to see if we can get to, uh, helping everyone with that. But early days. [00:24:34] Phil: cool. [00:24:35] Darrell: Yeah, I love that. And I, I, uh, so I was ex I was excited when, uh, Phil told me you were coming on because I had actually done a little bit of this at the basic level. Um, my friend and I were contemplating starting a business and we wanted to launch a program to help healthcare professionals transition into like remote knowledge, work type [00:25:00] jobs. [00:25:00] And I wonder if you could give me some maybe critiques on as to our process and maybe help us expand [00:25:07] John: Hmm. [00:25:08] Darrell: what the type of of questions that we were asking. So one thing that we started with, and I think this is gonna be helpful because I think marketers are doing the same thing. They're taking their customer and then they're asking these types of questions. [00:25:20] The questions that we asked, and we did it through chat, GPT and is, Hey, pretend you're a nurse you've been working at as a nurse for. You know, three to five years and, you know, you're contemplating a career change. You know, what are your biggest fears about changing a career? What don't you like about your current work daily, you know, day-to-day and, you know, if there was a program to help you go from where you are now to a well-paying remote job, how much would you pay for that? [00:25:59] You know, [00:26:00] how much would you actually, you know, part with your, your hard earned income for? Are we, am I thinking the right way? And you know, what other things do you think that, you know, if you were giving advice to marketers, for example, would you go deeper? Is that a good place to start? How would you kind of approach [00:26:20] John: Yeah. [00:26:21] Darrell: these questions? [00:26:22] John: Yeah. So, uh, a couple pieces. So one is that we, we tend to start with just. Here's a single entity and represent all nurses, right? So, um, so the sort of layer that we're really moving to now is, rather than having a single representation, let's look at the distribution of personalities of, of different traits that are just in humanity, regardless, sort of cult across cultures. [00:26:45] And let's at least represent some of that diversity of like, people who are more open to change, people who are really nervous, people who are much more outgoing, people who are, you know, ha maybe have a certain family situation. And so I think that those, we can [00:27:00] actually try to represent a little more of that breadth and then say, okay, of these, and, and remember there's no. [00:27:07] Penalty to having like a thousand trials on this, right? With a thousand different, um, well, so, so it's just, there's a interesting scale thing that we have to think differently about, and that's where the large brands and the most sophisticated users of these kind of things are starting to think from that perspective. [00:27:25] So that, that's one, that's, I guess the one thing. The other is just never hesitate to keep asking the why and then underlyingly what, what's behind that. [00:27:35] Darrell: Hmm. [00:27:36] John: and so the, the one question I heard that I'd be really concerned about is like, how much would you pay for this? That's a classic thing of grounding where the [00:27:45] LMS just don't have a concept of, of, you know, time, space, money. [00:27:49] So these are magnitude questions, right. And they're not good at that kind of thing. Um, so I wouldn't really, while you could probably have what was written on Reddit and Facebook and elsewhere, um, [00:28:00] it, it really wouldn't be good at that kind of judgment. So, um. When, when something tries to time one of my classes, it's like miles off, right? [00:28:09] And so it, it's just like, um, anyway, so, so I guess I'm, I want you to think both that it's probably quite good at that sort of qualitative nugget. I guess I, you have this luxury of being able to have many different circumstances that you could imagine. So why, what are, what are the range of reasons why people might wanna leave nursing? [00:28:28] And then let's try, you know, and it could be family reasons, it could be the time, it could be burnout, it could be emotional things. So, so let's try to represent that range and then for those target audiences, what are the things that are, are the motivators and go from there. So I think, I think it's just, you can do. [00:28:46] This is the challenge is that you can be very like scaled and sophisticated in what you're doing. And the other side is if you had a con, you, you can do a cycle here, right? So we can do, um, okay, based on, based [00:29:00] on these things we're finding, here's six possibilities of how we could structure this course or frame this, the messaging. [00:29:07] Now let's go back and test them with that same audience. Okay? And here's the responses they got. Now let's do that again. Now let's do that. So you can do this interesting cycle, right? Where we can have an innovation cycle that's synthetic. I know it's totally creepy, but logically you could have what are the things that bubbled at the top there and why? [00:29:23] And then we could start to see is that representative what we're seeing with the real humans? And then see if that's actually beneficial. 'cause then we have more off the wall crazy ideas. We try. 'cause again, they're sort of free quote unquote because it's not hurting us to, [00:29:38] to just try it in the synthetic world. [00:29:40] ​ [00:31:35] Phil: like ideally in a perfect world, we have a lot of actual human research data to, to get started, right? Like that level one of, you know, training and prompting and, and telling how to answer certain question, like you said, like it, it's not gonna get that grounding stuff, the pricing stuff, like a lot of different, uh, components there. [00:31:56] So when we start with a bunch of existing data from actual [00:32:00] human research and we pair that with behavioral data that we have, right? Like your talks a lot about the behavioral data. Marketers have funnels of data on every step where people drop off, and the pricing page experiments that we've done in [00:32:14] the past. [00:32:15] So let's say someone is [00:32:16] listening right now, John, and they're like, we have a lot of this data somewhere and a bunch of [00:32:20] different data warehouses. We also have a product team that did a bunch of qualitative research. That, that data is kind of everywhere. It's, It's, [00:32:28] not an essential spot. How do we bring it all together to create this like, live persona of our, like a hive mind of like our, our users there? [00:32:37] Like, um, can you walk us through, like, I know you talked about like agen components to this. I feel like that like the first layer to the last layer is where you kinda lost me a little bit. Like maybe unpack that a bit [00:32:46] more. [00:32:47] John: yeah, no problem. So, um, and, and the truth is that there isn't one right way to do this. It's such early days, [00:32:53] right? [00:32:53] So we're, we're, all learning. Um, uh, so, uh, the situation you've got is very [00:33:00] common. So we actually get asked a lot like, Hey, I've got these 500 interviews lying on the shelf that are only, you know, 18 months old. [00:33:06] What did we, what did we leave in the cutting room floor? What are interesting insights that we didn. Think to pull out, 'cause that wasn't the focus of this particular study. So, So, actually looking at them from that sort of fresh set of eyes. Um, and, and that's something that these, these tools, given the right context or the right framing, so you can try to make them expert ethnographers or, or anthropologists or, you know, kind of looking at it from these multidimensional facets. [00:33:32] And remember, I can have even just one interview and have one look at it from a accessibility perspective, another from a marketing perspective, another from, so it's just. And it, and we'll just burn up Every token philanthropic has to offer if we do everyone right, so have to be a little selective. We're very good at burning through tokens here at brilliant experience. [00:33:49] But, um, uh, but, uh, you know, it's a really interesting range that we can do. And so to your point, logically I've got, um, decks that are sitting around. I've got maybe [00:34:00] secondary research paper PDFs from Gardner or Forster or something. I've got, uh, and maybe custom research. And then I've got my, um, just our goals for, for our quarter right now and so on. [00:34:11] Um, and maybe these potential product things. Not only that, we've got all this marketing data, then we've got the product innovation teams data. And logically there could be people, if I've got a SaaS product or something where I've just got like usage data, right? So there's so many, and then there's social media signals, right? [00:34:26] So I'm just like drowning in stuff that I'm under utilizing. And so there are tools. Um, a good example is there's one and I, I'm sorry, I'm just. Giving you representative examples. There are a lot of these, but there's the one that's really unique is called next. And they, um, uh, next app Doco. But they, um, allow you to basically use, um, agents to look at everything someone's doing, like maybe an open comment or the number of times they get to customer service or their social listening and tries to route [00:35:00] these things to, here's something that's relevant. [00:35:01] So let's take, um, one of their clients is Bosch. So the people who do dishwashers, here's something that we heard about this for the marketing team. Here's what we heard about this message. And so there are ways to try to have this sort of giant data funnel, um, that people are trying to do that are pretty interesting to keep up with the, the world. [00:35:21] Um, so exactly how we do this in our, in our sort of multi-step ag agentic process is non-trivial. And I don't know if I can give you an accurate thing in our, in our hour, but the notion is how, how can you best. Distill the most important pieces. So obviously we just blow up context. If we have 500 interviews or, you know, thousands and thousands of, um, um, survey, you know, responses and probably hundreds of thousands for many people. [00:35:50] Um, so ultimately, what are some of the distillations of that that we can put together? And that's, that's what we're trying to do is say of all the noise, like today, I said, [00:36:00] Hey, I've got my coffee. I'm ready to go. We don't need to know about that. We need to know about the ENT stuff. So how can we, how we take that out of the equation and focus more on the other pieces. [00:36:09] And so that's where we're trying to not lose the essence, but we're also, we just have to make it more focused. Um, so there's, you then get into like vectorizing databases and coding and all these other nerdy science stuff. And I, and I guess I wanna say that the concept is an interesting one and it's open for debate how really to do this best. [00:36:30] Phil: Gotcha. So a good teaser of what your course is actually unpacking, because if, if someone signs up for your course on Maven, like you're walking non-technical users through the technical process, here is the, the terminal, how you actually do this, and we're gonna synthesize all this and like mark down files so that anthropic actually [00:36:49] understands, like [00:36:50] that's, um, [00:36:50] give, give us a teaser of the course, [00:36:52] if [00:36:52] you [00:36:52] John: Yeah, yeah, sure. So I mean, the, the idea, so there's really two sides, right? I've got one where it's like, let's use these SaaS tools and really understand what's possible today [00:37:00] in that world. And that's really important too. And then, um, but inevitably people are like, well, but I have this special use [00:37:08] And so, so that's what this is about, is starting them with, Hey, let's just analyze interviews or, um, our, our classic survey data or a combination. So what would be the steps we would take to, to synthesize that together? And then they've got their tools. We can go off to Qualtrics or we can go off to, it might be, um, uh, there's a thing called Marvin that people use for bringing a lot of, of qual data together. [00:37:32] For example. What, what comes outta those is what we're doing here. Any good relative to that? And how can, but the thing that's different, I guess, is we try to get you to think about what would be the stepwise process, right? So. With the SaaS tool, I give you the stuff, gives you the answer, but you might not be able to ask it questions. We don't really know what happened in between, right? There were like seven steps that we aren't privy to. They're private stuff. Could we actually replicate it so we can see what's the first [00:38:00] byproduct? So if we have 30 interviews, let's start simple. Um, what would be the summary, the key points that are extracted out of those interviews and what might be the, um, persona characteristics that we'd pull out of each interview? [00:38:12] Then so, so we've got this byproduct that we can look at and what was the quality of that? Then from there, we can say, okay, of that, what might be the archetypes that we're getting or the, you know, or the segments that we're getting out of these 30 interviews. And then we can see what, what did it do for that and why? [00:38:27] And so I guess what I'm saying is at each stage you can see. The magic and then tweak it and make it better for you and make it more trusting for you. And also logically, it doesn't have to be just you, Phil, or Darrell. It could be, I've got my AI agent that's the expert and, and reviews and critiques and gets it to do a cycle. [00:38:46] So we can, at each stage, we can say, uh, for example, if it's pulling out quotes, let's have one thing that does nothing. It's just a dumb system that says, is this a real quote, yes or no? And fix it before we get any further so we can trust the [00:39:00] data so we don't have to worry as much about hallucinations and things. [00:39:03] So interestingly, all these things that you're nervous about, we can see, you know, step by step how the reasoning is done. We can see what it actually produced is that byproduct. Just like if we had a real junior person doing something, it's like, well, gimme your draft. Okay, let's see what it's like. Let me give you some critique. [00:39:19] And so the better we can do at first. So the first cycle might be kind of crappy for those summaries. Great. Let's give it a critique. Let's get it change its analysis system with that. And we do that. And this is called a vowels. It's a classic thing you do in all of the agentic world up to really, um, technical systems. [00:39:39] Um, and from there, if you can start to have fewer examples of good ones and bad ones, then you can let your agentic, um, evaluator keep critiquing and keep refining it as well. So you wanna get this sort of, um, I don't know, positive loop of just keep getting better and better and better and better as it goes through. [00:39:59] And so this is the [00:40:00] interesting thing, is that logically you can tune this to self tune. And then there's the magic, right? Um, 'cause really if, if you, if I ask either of you, Hey, I've got this interview, what are the like three insights you'd pull out? Okay, well what is an insight? How do I find one, what do I look for? [00:40:16] What are you talking about? And so there's a lot of that that is so. Ingrained in our knowledge, especially as experts, that it's hard for us to share. And so actually having it sort of tease out what it is we're seeking by seeing these byproducts and, and reviewing, I mean, I, I joke and say what I do is radical common sense, but, um, uh, you know, but logically how much do we wanna have, um, sort of secondary, um, products on the way to our final report? [00:40:45] And also who do we want reviewing it and in what ways? [00:40:50] Darrell: Totally, totally. [00:40:51] 5. Is the Average Persona Dead? Scale, Diversity, and the World Model --- [00:40:51] Darrell: One thing I wanted to ask you is, if it's now possible to pressure test 20 different market segments simultaneously in a virtual room, is there no more use for the average persona, quote unquote, and you know, how is your live synthetic user not just, you know, the new average persona? [00:41:12] John: Yeah, I mean, well we did our best with what we had, right? So, um, I think, I think the way to to think about that is what's tractable for one human brain and a hold in our head and try to work with as a marketer or other other people. And so this is where our of these other systems can have a hundred of these living in their head. [00:41:30] And so it just makes sense that in this prefecture in, in Japan, there's gonna be a different mindset than there is, um, in this place in Australia in comparison to this place in Chile. And so we might as well, you know. Let's try to be more inclusive and more, um, actually really listening to our target audiences and what signals do we have in all those places in the world. [00:41:52] So, so some groups are actually really trying to have live signal detection, be it, you know, um, things on Twitter [00:42:00] or things on, um, Instagram and so on. Um, what's happening in different parts of the world, sort of up to date. And so logically how are we seeing differences between them or interesting patterns that span across them. [00:42:12] And so how might we use that sort of dynamic data, especially when it comes to market trends for, say, product kind of, um, ideas or innovations or, or clothing things or, or things in the sort of consumer realm. So, so I guess my, my answer, Darryl, is that logically we can sort of create a world model and then what, what does, how does the world speak to this concept and how we might, we refine it? [00:42:36] So that's where you could essentially have hundreds of little studies, right? That we're doing. So globally and then collectively, what is that telling us? Um, so it's just a more, it's the same process, broadly with a more sophisticated analysis. 'cause we'd be overwhelmed or we'd need a huge team of humans to do this. [00:42:56] Phil: Very cool. Yeah. I'm, I wanna go back to something you said earlier, [00:43:00] uh, John, about like, [00:43:01] 6. Asking the Uncomfortable Questions: What AI Agents Reveal That Humans Won't --- [00:43:01] Phil: being able to ask inappropriate or like strange questions to the synthetic users. when you're you're doing these and leading studies, sometimes like someone says something [00:43:11] and just like each want to dig deeper, but you're like, ugh. Like, I don't know how open that person would be to. To, to answering that question. Um, it's a fun one. You know, like what is the most uncomfortable truth, uh, about a customer's deepest fears or blockers that, uh, one of your agents was able to, to reveal during the process. [00:43:30] John: Why? Wow. Um, yeah, those are pretty crazy. So, um, I mean, I, I think, uh, some of them come down to the, um. [00:43:37] Well, I wouldn't do this, but my, you know, wife always twist my arm. You know, those kinds of things where it's like, I, I can't admit to anyone, but I'm a slave to blankety blank. And so, so, you know, lots of times it's fears, right? [00:43:49] That you have or, um, or they're like, yes, I, I grew up poor, so I absolutely need to, like, I'm really thinking about money and I can't help myself. You know, like, so there can be very rational things there too that are not [00:44:00] just silly, but, but, um, I think that's a, a piece of humanity is we have these sort of built persona, you know, broad way we live, we evolve, but. [00:44:11] We have, you know, tendencies. And so we need to know what those are for the humans making decisions because they're gonna be drivers for us. So, um, so getting at those, um, the other thing, actually, lemme just turn it around a little bit. So we do a lot of stuff with, um, AI moderation of interviews as well. [00:44:30] So it can do not only just saying a question, but then giving you some sort of dynamic follow up that's related to what they were talking about. And there's a whole nuance to making that good, uh, or better. Um, and it's not perfect. We, I think US experts could always do a better job, but if you can get. 85% of what we do and then have it in Finnish and in Brazilian Portuguese. [00:44:50] Maybe we're doing something unique. But the thing I was gonna tell you about that is that people have this tendency to be more willing to open up to an AI [00:45:00] moderator for like things about their finances or things about their insecurities or about their medical situations because it's sort of neutral. [00:45:07] Then it won't be like, my God, you're that you, you have that situation. You're still eating Twinkies. Like, you know, you know, so you, it knows that they know it's sort of neutral even though they also somewhere in their head know that some human is gonna go look at these things. So it's a really interesting phenomena where we can get actually some more openness without us expert moderators in the way. [00:45:28] Phil: Very fascinating. Go for it, [00:45:30] Darryl. [00:45:30] Darrell: No, I was just gonna say that really resonates with me because I feel like oftentimes we don't buy something or we don't express interest in something because of social pressure. So you just feel like, you know, like let's say you wanna buy this certain kind of car, or you think this sweater is really cool and someone asks you, why didn't you buy that? [00:45:47] And the true answer was. You know, my wife thought it looked dumb or, you know, my friends like made fun of that car. So I really liked it, but now I didn't buy it. And, you [00:46:00] know, I, I wonder if there's, you know, this, we haven't talked about this yet, uh, John, but you know, when you're doing all, all of your, um, um, you know, working with clients, are there like common commonalities in the insights that, you know, like broad commonalities in the insights that you, that, that you learn from having them go through this synthetic, synthetic user testing? [00:46:23] Like, something that's, that's always kind of comes up like, oh, we were thinking about this wrong, or, oh, um, you know, it looks like we need to pivot this way. I mean, there might not be anything, but I, I just, I just thought that maybe there would be some things in common that you've been seeing. [00:46:40] John: I mean, I think one is the, um, the sort of ex sort of excitement. So like, you know, we all wanna be properly influencing our, our big goal is to influence major decisions, right? And, and have good, grounded, um, insights for them to make those calls. And, um, I think one of [00:47:00] the interesting things is how engaged they get in these synthetic users in comparison to a deck, right? [00:47:06] So, or whatever we present or whatever workshop we do, um, it can't really compete with sort of bring the data to life as it were. Um, and for that, it's even just a text chat right now, but logic could be speaking to that person or that representation or that set of representations. Um, so I think that's one thing is that the. [00:47:26] So now just to give you a sense, um, what one commonality is, um, you know, in 2022, we remember way back then we used to do our qual stuff. Would take a major global thing, would take about seven weeks. Now we can do it in, you know, roughly 10 days, you know, or, or in, and most of that time is synthesis and really think humans really thinking about the data and digging into it. [00:47:49] Um, so we don't drop that. But, um, uh, I, when I ask my, um, target clients, you know, we can do the classic 2022 style or we can do something [00:48:00] where we're having these synthetic moderators and, and, and then, and create either a deck or a synthetic user. As a result, just 90% of them want, why would I wait seven weeks when I can get something [00:48:11] in 10 [00:48:11] days? Why would I want a static deck when I can, when I can also have this thing come to life? Um, why wouldn't I want to have you look at their past data and have it feed into these thoughts? [00:48:23] So, so I think that, um. Broadly, that's, that's one change that's happening. I, I guess it's also as we teach people, uh, you know, Dar I think early on you said, why aren't, why isn't there more use of these synthetic users? [00:48:37] When these teams start to see what they can do, they're like really engaged. And then they start to dive in more and wonder and then say, oh, well if you have a, a Syntech user of this, why don't we have one in, in this segment as well? And there isn't a compelling reason. Why not. And so again, they, they start to see this like mountain of data they could [00:49:00] actually use in their, in support. [00:49:02] And so there aren't good tools yet for making that happen. And I think, you know, failure, you brought up, um, you've got this data from product land, innovation land. You've got this marketing data and data lake, and you've social media, and then you've got things that maybe just, um, innovators or some other group has got, and there's. [00:49:21] These divisions are silly now, right? We need to really try to bring together that knowledge to make these things more powerful. [00:49:29] Phil: Super cool. John, [00:49:30] 7. Ending the Quant vs. Qual Debate with Statistically Relevant Qualitative Data --- [00:49:30] Phil: I wanna ask you about, uh, statistical significance in this whole synthetic user, um, persona, like the life persona that we, we can chat with, like, um, having been in the lifecycle product marketing world, a lot of listeners are like in email marketing land and we deal a lot with working with the product team. Like, hey, new features coming out. We work with the marketing ops team, create a ticket. I wanna send out a new campaign. Oh, what's the message gonna be? Well, based on like chatting with three users, this is the qualitative feedback, we should message it like this or message it like that. Or the good product marketing teams have like a good messaging map and like, um, we should segment the messaging based like this, and the job to be done is different, blah, blah, blah. [00:50:12] Um, but like. There's always gonna be this debate in tension between the quantitative marketer and the qualitative product marketer messenger. And we often get into this debate about like, alright, what is is more valuable here? Getting qualitative insights from a handful of people about this message that we want to send out. [00:50:32] Or do we just run an experiment, like let's blast out two versions of this message to like a subset of of our audience and let the data tell us who is gonna win instead of relying on just, you know, a handful of those users. [00:50:46] I feel like the whole thing that you're building around synthetic users blows up the debate and like introduces a third aspect here almost. [00:50:54] Um, you've actually said that like, AI now allows for statistically relevant qualitative data [00:51:00] forever ending this age old like, um, debate between quant versus qual. Can you unpack that for us? [00:51:06] John: Yeah, well, I, you know, the, first of all, I'm, the sad truth is I don't have the answer for you today. Um, so let's really get that out front. But, um, uh, a couple pieces. One is that, um, with things like, like AI moderation is a good example. Um, we actually can do qual at scale, right? So then those two teams might be reasonably satisfied together, uh, if we're doing some sort of experiment. [00:51:29] So that's not even getting to the synthetic users, but I think, um, this is where it's really simple. We can build the best, you know, you can go say, Hey John, can you build us a set of synthetic users? That's. You know, represent our data. Great. I just ran this quant test over here. Let's go run it with your people, you know, your synthetic users, and see do we get the same stats or not? [00:51:51] So, so I think that this is where we really do want to put our money where our mouth is and start to say, not just direct, you know, we're starting with [00:52:00] directionally, are we getting something interesting? I think the answer is yes to that. So to your latter question, do we know if it's gonna be a 2% or a 3% lift? [00:52:09] No, we don't know yet. And we've gotta really try it. Um, but major groups are really, really trying to use their, their sort of, and think of it again with statistics. What are we doing? We're representing a population. What are these synthetic users representing a population? So it's, it's just another. Uh, it's almost a way it talks back to you, but essentially it's trying to represent it in a, in a, in a statistical way. [00:52:33] Right? That's exactly, it goes back to like neurons, so I don't wanna get too nerdy for you, but really this, the machine learning stuff on early days was really all about can we represent the way that our neurons in our brain fire and therefore, can we have something that has synaptic, you know, relationships? [00:52:49] And so some of them form stronger, some of them die off. Why is that? Can we find those patterns in some sort of algorithm? And so we did, and neural networks. And so neural networks are [00:53:00] sort of the underlying principle behind all this machine learning. And then, and then, you know, natural language moved us to ai. Phew. [00:53:08] Darrell: I wonder, is there like some pre-work that needs to be done when you are going to be rolling this out? In any department, you know what I mean? Because I wonder if the people's first reaction to, Hey, hey, everybody, we're gonna do the synthetic user testing is kind of like very skeptical, very like, do you do some convincing upfront or do you have your students do convincing? [00:53:33] Like what do you recommend to make sure that you can kind of get this done smoothly and start without so much resistance? [00:53:39] John: yeah. Uh, a really, really good question practically, and, um, you know, me too, like, it took me probably three or four months to not hear, you know, nails in a chalkboard when I said synthetic users. So, um, so I'm totally with you on that, and I, I've gone through all the emotional stages of like, oh, do I have a job anymore? [00:53:58] Is this any good? [00:53:59] [00:54:00] And so there is a piece to this that's about our underlying, like, value, value in the company, right? If this, if these things could be true, then what, what does that mean for us? So there's a big. Question. And actually we happen to be in a, in a place where, um, like yesterday, Jack Dorsey fired a whole bunch of his staff 'cause he thought AI could do more things, right? [00:54:19] So this is a real fear for practically for our, our people in marketing in, um, other things like that. So, so, um, I think the first thing is just helping people understand logically this is a tool. And so we can go from being, um, sort of nervous about this thing and un unsure to having some experience with it, decide when is it appropriate, right? [00:54:43] So I'm sort of building my way up. Um, so what we do is we very gently have a small group try these things. We often do a comparison between here's the real human data, here's the. Simulated stuff and we get that team to analyze it and decide what worked, what didn't [00:55:00] in their field, and then share it with their colleagues. [00:55:03] So it's not me professing it or you know, it's, it's in real life. This actually could have worked in this circumstance and not in this other circumstance. So very pragmatically. And then sort of easing it in to large other groups. 'cause there's the group that are gonna be like, this is great, let's run with it. [00:55:19] That's like 15%. There's about 80% that are the, just tell me what to do. I'll do it. And then there's the people like, no way. I, no way. Never. I'm, I like horseshoes. I like my VHS tapes. I am, I am not doing Netflix and I don't know what to do. You know, those folks, I just, I want them to really think carefully. [00:55:38] Um, cause it does sound creepy, but, you know, you were willing to use stats, you learned stats, you know, probably in, in university. So this is just another tool that has that, it's just, it feels creepier 'cause it's trying to truly represent what the person would. [00:55:53] Phil: Um, John, I, I'd be remiss if I didn't, uh, I [00:55:55] know we're running up on time, but I, I want to ask you, uh, a question about your book. Like [00:56:00] I, I, I, I told you I had of this, that I, I [00:56:02] got a chance to read your book design for how people think. Um, it was a fascinating movie, I think a ton of applications for, for marketers, product marketers, lifecycle marketers, um, that, you know, are always [00:56:12] obsessing about why people are doing certain things. [00:56:15] Like [00:56:15] we see the behavioral data on [00:56:17] our funnels, but we don't really understand why someone dropped off in the pricing page or why someone decided to unsubscribe from that email. We can't even reach out and [00:56:27] ask that person, like, we don't have that data. Is that that a good use case for the synthetic users? Like in the whole MarTech world, like that behavioral data doesn't answer the why. Um, [00:56:37] 8. Mining the 'Why' Behind Silent Behavioral Data with Synthetic Users --- [00:56:37] Phil: do you you think that like interrogating 500 hours of past interview data through the lens of a synthetic user allows us to like uncover the why behind some of this like silent data inside of our funnels? What Are your thoughts there? [00:56:50] John: Yeah. Uh, so interesting. So, um, uh, first of all, I mean, I think representing what people need and how they think is even more valuable now that we've got these co thinking [00:57:00] tools. So for product innovating, that's a really interesting piece of, I think of, I didn't, I had a, frankly, a, a mediocre chapter in my book. [00:57:07] That's the only one. Um, but, uh, that, that really, um, really said, Hey, we should think about this with AI and, you know, stay tuned. And that was, I don't know, 2017. So, you know, a lot's changed. [00:57:19] Um, so, so yes, version two is, is. W is necessary. However, um, I think that, um, this is a really interesting way to ask those impossible questions, exactly what you're talking about. [00:57:31] So if we wanted to know like, what do like super extreme rock, um, rock climbers need, or what do, what do these people who dropped off at the funnel, or what about extreme high net worth people who are never gonna take, you know, a $300 incentive, why I don't need that money? Like, why would I show up? So, so these groups that you can't reach out to at all are a great example of, let's get some thoughts about what might be the real possibilities here, and then what evidence do we have to support or, or against that. [00:57:58] So think of this as like [00:58:00] converging lines of evidence. What else do we have and is this compatible with that? So, So, yeah, treat it. I, I guess I asking everyone to have. Some honest curiosity. Really try these things, understand what's possible, and then be skeptical. Don't just take it and run with it. I've had sort of junior designers say, this is great. [00:58:17] I'm gonna use this right now for X. And I'm like, well, just hold on. Um, let, let's see. You know how good it is so you know when to trust it, right? So, so I think there's a balanced way to do this and, you know, the people who are doing heart surgery tools versus the people who are doing like potato chip advertising, well one of those can go out on a limb a little bit more than the other, hopefully. [00:58:37] And so, so it's gonna be different appropriateness in different situations. [00:58:42] Phil: Very [00:58:43] Darrell: I would say like the big, so if I was like a, I do this like skeptical executive exercise, so I would say skeptical executives would probably say, Hey, this is, you know, now you can do 20 markets, same, same time, whatever. [00:59:00] Does this lead to like analysis paralysis? Because we're spending so much time designing the user, stu, user studies, the synthetic user studies. [00:59:09] And then I think the other skepticism would be like. This still doesn't put, this is still, isn't where the rubber meets the road. We're [00:59:17] still theoretical, you know? So how would you address those two quite, this is not what I think, by the way, [00:59:24] this, [00:59:24] John: That's [00:59:24] Darrell: this, this is what [00:59:25] John: They're perfectly rational questions. Yeah, So, so the fir on the first point, um, you know, this is where I try to train people in my classes to not just, um, to be orchestrators of systems, right? So the orchestrator like, wants certain quality, wants to, they get to pick who plays oboe. [00:59:42] They get to pick, you know, how you're gonna play it. And so we want to actually build a, a unified system. So if we're doing these 20 markets, it shouldn't be that I, you know, I now need 20 people to do each market. It's that I have my synthetic orchestration system, you know.[01:00:00] [01:00:00] Cranks these things up. Now I am the person who is the quality, the human is the quality control person. [01:00:06] I wanna make sure, is this accurate of what we know of that audience is this, you know, so we're judging its quality. Um, and then to your point, these are what, um, forecasts or hypothesis or, or, you know, so that you're absolutely right. These are not real humans, you know, slapping down their credit card or doing whatever behavior. [01:00:25] So you're absolutely right about that. So I think that's where we need to have, you know, almost every group goes through a stage of what's something we know really deeply. Let's get, uh, a team. So a a lot of the groups, when I interview the founders, they say, yeah, what they do is they give it, get us to, um, give them a study that they've already done with real humans to see how close they map, right? [01:00:48] And then they have a sense of, oh, you know, and they don't, obviously don't share with the synthetic user team what the findings were. They, they, you know, collect it first or keep it. Um, so the [01:01:00] question is, is this actually an accurate predictor? Right? And so, to your point, um, there are a couple interesting things. [01:01:05] So one is, um, first of all, it's not good. How, how can we tweak it? How do we train it? And so again, this is, is a, a cycle that one of the things that people have found actually in, um, uh, doing so, so going back to the quant and trusting our survey data, right? Um, there was a group that we met with and they said, gosh, you know, um, we, you know, we get about, I don't know, 80, 75% right on, on these surveys based on what we think, you know, humans would do. [01:01:36] And then if we layer on top of that, there's gonna be this proportion of bots and this many people phoning it in for the money. Then they get like 95%. And [01:01:46] so, so my point is that actually we have these imperfect tools already, right? And we're, we're, we've got human frailty and we're not, you know, uh, getting into the pineal gland of your Bishop Berkeley or going to the matrix and plugging in. [01:01:59] So [01:02:00] we can't, we don't get the direct source. Um, and so, so actually that, that's an interesting imperfection of any of our tools right now, except for say, like, did they buy or not? [01:02:10] But, um, uh, so it's a really interesting challenge where actually we might be, um, over accurate in, in logically in a, in a representation of what the human would do. [01:02:23] And then our measurement tools are imperfect and we have to account for that. So that, that's a logical possibility maybe on the other end that we may not have thought of. [01:02:31] 9. Designing for Agent Users: The Coming Shift to Human-and-Machine-Centered Design --- [01:02:31] Phil: You mentioned bots there. Uh, John, we've talked a lot about like agent to agent commerce on the show or machine customers, people call it, uh, differently a little bit. Um, but you know, your book talks a lot about designing for humans. Your synthetic users are essentially like simulating an environment where we are talking with humans and, and like a persona of our humans. [01:02:53] What happens to a world where we need to shift and [01:03:00] start thinking about how do we talk to agents that represent humans? is it a massive [01:03:05] shift? Is it just like thinking about it a bit differently? Um, but I know you're predicting this like huge advantage for teams that are taking the shift seriously. [01:03:13] Specifically those that are like designing for both the human and the agent user simultaneously. Just curious your thoughts there. [01:03:20] John: Yeah, I mean, I, I, the truth is that I don't really deeply know enough about, uh, the agentic tools for research or that might be helping to influence it. Um, you know, so if, if I've got, um, oh gosh, some, some of the, um, adjunct tools, like on my browser, I can click something and then say, Hey, go find me this product on this website, right? [01:03:39] Um, or I, I have some of the specialized browsers that have just the, the agentic, you know, system or for that matter. Um, so for any of those, I think we have to study and those are gonna be really moving targets of how they approach the problem. Certainly gonna be algorithmic. Those groups are gonna want to use as few tokens as possible to get the best answer. [01:03:59] They're going [01:04:00] to, you know, they can read things that are all white on white, you know, so, so there's huge differences, I think, in how we design for bots. And so, so yeah, it, while. The underlying thinking. Actually, I encourage everyone, regardless of what you do, even if you never touch these synthetic users, but I think you should just listen to hit that Thinking triangle or Chevron and actually watch the, the thinking process of the agent as they're doing things. [01:04:30] You'll learn so much about the approach that they're like, oh, I think the user wanted this, but I gave them that. That might be, that might suck. Maybe I should think about it a different way. And so actually seeing that little line of reasoning couldn't be really helpful to, to guide that. But I I, another piece is a little bit of the reverse engineering, right? [01:04:48] We're just gonna have, there's gonna be on both ends, there's gonna be people trying to tweak so that they get advantages with bots. And the bots are gonna do their best to not be tweaked. So there's gonna be a cat and mouse game for [01:05:00] sure. I mean, I am sure we all have that thought experiment. So that's why you need your actually futurist, um, agentic workflow to think through these future possibilities. [01:05:10] Phil: Very cool. John, we've, uh, we've kept you for a while here. Really appreciate your time. This has been super fun. Fascinating applications for marketing ops and, and lifecycle folks here. I know you're, you're deeper in [01:05:21] the, the product side of [01:05:22] things, but Darryll, you want to take us home with the [01:05:25] last question? [01:05:26] Darrell: last question. [01:05:28] 10. The Happiness Question: Dogs, Nature, and Staying Analog --- [01:05:28] Darrell: So John, you're a teacher, a consultant, a founder, a cognitive scientist, an author, a speaker, but you're also a dad and you're also a dog dad. [01:05:38] One question we ask everyone is how do you decide what deserves your energy at any given moment? What's your personal system for staying aligned with what truly makes you happy? [01:05:48] John: Wow. Um, yeah, that, that's a work in progress always. Um, but uh, yeah, I think over the years I've learned, um. A to just celebrate everyone else. And, [01:06:00] and I actually think the dogs are a great example where, you know, every day comes in as like, oh my God, it's another day. This is awesome and this is a good reminder that, you know, oh, taxes aren't due for them. [01:06:09] Or, you know, it's just like they're just living. And so I think that reminding ourselves to be that simplistic about the world sometimes is great. And, and all of those, um, groups I really care about, like taking them into nature and getting myself away from my phone and everything else is, is awesome. Um, I am one of the guilty dog walkers who listens to podcasts while we're doing that. [01:06:29] And so I always say that my dog, Lola and I were, you know, hearing about this type of agentic, whatever workflow, um, and she really wanted to hear it. But, but I think that connecting back to our analog selves, and we know from our being psychologists how crucial it is to just have a little bit of nature, a little bit of down, and actually that we can be so much more innovative by just giving ourselves that veg time to let the ideas bubble up. [01:06:55] So. [01:06:56] Phil: Love it. Great answer, John. We'll, uh, link out to both courses [01:07:00] you have on Maven as well as your book. But um, yeah, feel free to take the next two minutes and, and, and plug some of your other stuff or plug the course. I know we talked about it a little bit, but feel free to. [01:07:11] John: Well, you know, I, I think just generally everyone just, just keep trying these things. Testing the systems where, you know, I'm a researcher and we, we actually all have to become researchers to say what's really possible. Now keep, you know, there are so many people that say, oh, you know, I tried that, you know, six months ago. [01:07:27] I don't know. They're changing because things are changing at, at not just a linear fashion, but a logarithmic fashion. We know that humans don't understand logarithms, so, so we can't really, in a visceral way, understand how fast things are changing. So just sort of beg you to keep trying new things and keep testing them and seeing what's possible. [01:07:48] So, um, I think that curiosity but skepticism and so. If you want us to do it for you in testing things, we have a great consultancy if you wanna learn it. Uh, we've got ways to share that with you too, but [01:08:00] um, I just want you to do it for yourself as well. Um, and, um, yeah, I think that's it. [01:08:05] Phil: Awesome. Great advice John. Really appreciate your time today. This is really [01:08:09] fun. [01:08:10] John: My pleasure.