Subscribe
Share
Share
Embed
Get a seat at the table and build the design career you want. This podcast is for designers looking to break in, level up, and take control of their careers—whether you're freelancing, climbing the corporate ladder, or just trying to get noticed. Every two weeks, we dive into career fundamentals, design best practices, and the hottest topics in the design community.
Would you consider it to be a corporation or a large company? Or is it
Tyler:Where am I currently?
Nick:Yes. Yes.
Tyler:We're about like 60 people. I don't know if you consider that large.
Nick:I would consider it large, but not not corporate per se. The reason I'm asking is because like most of the research I've done, like really dedicated research, all was at the large. And we're officially live. We're officially live. Exactly.
Nick:I mean, this is just an experiment. And, you know, just a while ago, something happened in a design project that really made me unsure about everything I've learned so far. We were doing research, user research just to validate and check with a prototype, and we were talking about screenshots. And at the end of the prototype, we ask the users or or the participants to tell us how it went, you know, all the standard questions. And then at some point, we asked how useful or not useful were the screenshots for you.
Nick:And almost all users told us, yeah, they were great, very useful. It really helped me achieve my goals, like all the, you know, very positive answers. But looking at the recordings, some of those people didn't even visit a page with screenshots on them, and others were just scrolling by so quickly that it's impossible for them to really look at the screenshots and the little arrows and annotations within the screenshots. So that makes me sound a bit like, you know, doctor Gregory House from TV show. You know, all people lie.
Nick:And that makes me wonder, should I mean, why are we doing research? Why are we asking these questions if people are just making things up, if they just want to be friendly because they know they're part of a a research program? And what's a better way? What do you think?
Tyler:Yeah. I think it I think picking the type of research for the type of thing you're testing is also key as well. So in your example, you mentioned a screenshot. Like, how do you asking them if the screenshot is is helpful. I guess you're kinda benchmarking the usefulness of these screenshots as, like, facilitating the experience.
Nick:Yeah.
Tyler:What might have been a better test might have been a usability test. So having them navigate through the experience and having the screenshot be the the helper. So if you wanna AB test two different experiences, one without screenshot, one with, and seeing if they get to their goal quicker. Yeah. And so I think it just goes back to the right I think all research is useful.
Tyler:It just depends on choosing the right form to to get the right feedback that you want.
Nick:Yeah. Yeah. That's probably true. Just a little bit of context is that we were testing different variations of the screenshots. Mhmm.
Nick:Because from our, you know, regular feedback collection practices, we already got comments where people like were like, yeah. Use screenshots. We're really helpful. And then we were like, oh, interesting. Let's see if we can make them better.
Nick:You know? So we were AB testing between screenshots, and then, you know, the outcome was that it doesn't really matter. Like, they're all kinda useful. But the thing that was really annoying or freeing perhaps for me was that as a response to a question, some participants or users will just make things up either to be friendly or because they feel like they are tested and they don't want to give a wrong answer or because we are asking silly questions. That might also have been the case, but it I I really scratched my head like, you know, maybe it's outdated.
Nick:Maybe I fell for some educational thing that only works in theory. But I wasn't in the best mood after after I encountered
Tyler:That's fair. Well, like, every like, I think it's important to to know that every test, whether it fails or not, there's always a learning opportunity. So in that case, maybe just another way to do it. But, like, the old saying, what users say they want something, but they actually watch them do it, they do another thing. What I think is what I often do is is try the triangulation approach to be to use a fancy word, to do multiple types of research for the same study.
Tyler:So that could be a combination of things. One could be a survey, and that could be use it like your screener upfront. Like, let's get some context on some some high level questions first, and then you'll run the same study you did, and then then you'll do something that's unmoderated. That way, you're kind of covering your bases from all angles. Like, you're not using one one data point, using three in this case, and you're able to kind of cover your bases.
Tyler:Like, here's what the the survey said. Here's what they actually did, and here's our in person kind of conversation. Let's try to weed out kind of or kind of narrow in on, like, the the key insights there.
Nick:Right. Yeah. That makes that makes sense. I do feel like listening to you and all the conversations we're having is that you are way more research focused than I am or perhaps more proficient in research.
Tyler:Well, it's it just happens. Like, I've I've been in the situations where you're frustrated, like, okay. You're not get Mhmm. You're not giving me what I need. But, like, I think it's it's about being empathetic again.
Tyler:Like, you're they're participating and, like, to your you mentioned that before, which I think you should kind of narrow in on. It's like
Nick:they're
Tyler:they're oftentimes, they're volunteering or you're paying them. So, like, they feel like they need to give you some value. Yeah. It was like the what the part of the process or, like, at least part of my process, what I like to do is to to make them at ease. Like, I preface every call with, hey.
Tyler:We're doing this type of research study. That's why we reached out to you. I would love to I would love for you to be as open as possible. We are looking for both negative and positive. So if this thing is shit, for example, please tell us.
Tyler:That's more that's more important than compliments. Like, we're looking for I'm looking for honest feedback, and I would love for you to kinda give me to that. Like, give me that kind of information. And then that usually, like so it's a lot of, like, conversation, getting my knees until they're in the right mindset to give you the information that they want. So that might that usually helps because, like, you you can tell that if they're they're not they're uneasy and also if they're there's kind of different types of people who join the sessions, and you've kind of probably seen this.
Tyler:There's a person who's giving one word answers and seems very stiff, or they just have that kind of, like, direct personality. Like, getting getting into, like, telling jokes or just getting my knees just to, like, break out of that that tunnel vision they have just to get them in the right mind frame will give you the information that you need.
Nick:Right. Yeah. That makes sense. Extra challenging bit of that situation I started this episode with was that we used a platform where we can ask questions, but we aren't live in the room with them. Mhmm.
Nick:You know, it's so it's all prerecorded, and they go through the prototype, and then they they we ask we show them questions, and we ask them to think out loud, and then we're recording still. So when they answer a question they see on screen, like, I cannot ask a follow-up question. I also cannot really put them at ease except by adding some text at the start of of the test. So so point well taken. I fully agree.
Nick:It didn't help in my situation in that particular moment. I think in that moment, and for the next version, we just decided to not ask any questions. We were going to go fully by, hey. Think out loud while you're going through the prototype, and then just watching them do the thing, And just then just seeing if they can succeed, yes or no, and where they got stuck in that kind of thing.
Tyler:Could be. But also, I would also think differently about, like, the moderated and unmoderated. The unmoderated one, usually, it's a volume game because there's instance, like in your example, someone's not actually looking at the mock up or the actual image. If the volume of that study increases to compensate for, like, the unmoderated, you'd have to have a higher volume count of participants versus the moderated. You get more you're there in the room.
Tyler:You get to kind of push back and ask questions.
Nick:Yeah. True. That's true. Yeah. Also makes me think about, like like, you're working somewhere.
Nick:Would you consider it to be a corporation or a large company?
Tyler:Or is it Where am I currently?
Nick:Yes. Yes.
Tyler:We're about, like, 60 people. I don't know if
Nick:you consider that large. I would consider it large, but not not corporate per se. Reason I'm asking is because, like, most of the research I've done, like, really dedicated research, all was at the large companies. And then in recent times, I've been more focused on, you know, startups and and founders and content creators. So research is far less common there.
Nick:So maybe I've neglected my research muscle a little bit. I'm just thinking about how expertly you sound when you talk about about research. So maybe that's your one of your super skills.
Tyler:Potentially.
Nick:Yeah. But yeah. I mean, it's super important still.
Tyler:It really depends on, like, the situation. You can probably skip research if it's a thing that exists that we all use. Like, if it like, how many times can you redesign or or usability test a checkout? Like, if it's
Nick:Yeah.
Tyler:If it's a pattern that exists that's repeatable, we know what the benchmarks are. It's it's easier. It's really it really depends on the situation.
Nick:Yeah. Well, that's true. Yeah. Why why reinvent the wheel? And also important is that people, they bring their experience with them to your platform.
Nick:I mean, that's the only reason why the hamburger menu is still a thing because Google is so large. They introduced it and made it common, and people just had to get used to it. So now you can be like, well, I I want to design it differently because it's better. But, yeah, it could be, but then it's not it's better, but people are not used to it. And then they get confused even though it's not sure why I'm saying that, but I just felt right in the right in the moment.
Tyler:I think we all love the burger menu.
Nick:Oh, yeah. Yeah. Yeah. For sure. Anyway yeah.
Nick:So so I was really a bit sad when all of that happened, and then I tried to convince my you know, the the person that that was involved also in the research of something I did a while back, a sus score. Does that ring a bell? Sus score?
Tyler:Tell me.
Nick:Interesting. Sus score, I might I I have to look at my abbreviation, the system usability scale.
Tyler:Okay. I was thinking something different.
Nick:Yes. It was sus as in suspect.
Tyler:Exactly.
Nick:Yeah. Yeah. Yeah. So you are a gamer. Anyway, so sus score is you you do a test and you ask 10 questions afterwards, the same questions all the time, and then there's a score, and there's then there's a formula that you apply, and then they're based on the answers to those questions, you get a score.
Nick:So it's really standardized. And then if you do that with, let's say, 10 people, you have an average. And then if you make a change to your platform, you do the test again, you get a new standardized score, and then that's an easy way to see if that if you have an improvement. Yes or no. Because the score will be higher.
Nick:And the reason why it's useful is because you ask the same questions all the time, you know, and you give it scores, then it's, like, it's it's really a numbers game. I thought it was quite useful, then the person I was working with on that research where the you know, about screenshots, I told them also about SUS scoring. They were like, yeah, well, that's just a spreadsheet you used to sell to your corporate manager, but still people lie on it. So, like, he was really convinced that these questions do not matter at all. I'm curious what you think there.
Nick:Like, when like, are questions useful or scoring or testing and comparing it, or is that really just, you know, paper jungle in a large corporation and and thus very useless?
Tyler:Think so. I think it's very useful. Like, again, like, I'm a advocate for multiple data points. That sus score seems very valuable, like asking questions and then seeing if the score changed. It's it's almost similar to, an NPS score.
Tyler:Like, just like Yeah. A health check. It's a a similar thing. Thing you you send to users or to your client base. Like, I wanna see if if we're doing better or worse.
Tyler:And because you have questions, you probably have more qualitative data as part of that so you can actually dig through people's comments and answers and seeing what they're saying
Nick:Yeah.
Tyler:Which gives you more information than radar software one to five. Yeah. But the added benefit, which I love, like, if you're really nerdy about it, is that you're able to I imagine you have access to the individual users' answers.
Nick:Mhmm.
Tyler:Yeah. So you can probably segment the data to go for if there is a score, like, one out of 10, or, like, how do you grade? Like
Nick:Yeah. Do grade? 10. Yeah. You you get a Yeah.
Nick:You get you get a grade, basically.
Tyler:What you could do is basically just segment anyone who gives gives you a failing score, so, like, six and below or five and below, watch over the year. If, like, if you're doing this quarterly or every other month, watch to see which users are going or which segment is going up or trending up and which ones are going down. And that kinda gives you an inkling on, like, if you're improving. And also, can isolate into the users who are not super super jazzed about your software. And those are actually great users to to interview.
Tyler:Yeah. I find, like, the the the angriest users are the best users to interview because they're angry because they care. Yeah. That's the only that's the only reason that they're giving terrible score because otherwise, they just give you a seven, move on with their date. But they've your software is has enough impact on their life that they're they're they have a net negative experience because they they're trying to rely on it for whatever task they're trying to complete.
Nick:Yeah. Yeah. Oh, yeah. For sure. Yeah.
Nick:Yeah. I've I've I've spoken to users or clients of a large Dutch bank, and then they were very annoyed, very angry. Like, they they in full emotion, they they told me, like, well, I've been a customer here for forty years now, and you still have to ask me where I live. I still have to prove to you who I am. How dare you?
Nick:You know, you've known me for forty years. I've given you all my business. They were very upset. You know? So, yeah, I can I can really see that?
Nick:Curious about
Tyler:yeah. In those instances, like, they just wanna be heard. Like, I've been Yeah. I've been shell shocked in those conversations. Initially, I was like, no.
Tyler:I'm I'm gonna be reviewing a four to five four out of 10 today. Like, I'm dreading those conversations. But what you what you learn over time is that they just wanna yell for a bit and then understand that you're empathetic and you're listening, and your goal is to fix whatever problem. That's why, like, in in most of those sessions, I never even mentioned that I'm a product designer. I just say I'm an extension of the customer success team.
Tyler:They're like, here, vent away, but I'm here. I'm also like, I I want you to tell me what's wrong, but also I have the ability to add what you're saying to the backlog so that our our engineering team can fix your problem. And then once they hear, like, oh, okay. Cool. And now we're I'll add, like, a a little kind of email.
Tyler:So thank you for the thing. We've covered this. Value your information. Do you mind if I reach back to you for future investigations? And then Yeah.
Tyler:Cool. I'm part of the product now. So it's you're getting them, and it's fun to see them, like, in your example, your SUS scores. Mhmm. You're able to kinda see if the you're they're that conversation moves them up in in terms of that rating over time.
Nick:True. Do you hide the fact that you're a product designer on purpose, or did that
Tyler:just happen? I I I only hide it because I'm I was tired of explaining what a UI UX designer was. I was
Nick:like Yeah.
Tyler:I am
Nick:So you make things pretty. Yeah. Yeah.
Tyler:Or they didn't even understand what that is. Like, you're what? I'm like, okay. Well, I'm customer success.
Nick:They get it. Yeah. Yeah. Yeah. Well, that makes the yeah.
Nick:I'm part of the team that builds your product. Yeah. You you mentioned data points, so I think that's very interesting because a poorly phrased question and an answer to that poorly phrased question is useless. You know? Did you like the screenshots?
Nick:How useful was it were the screenshots? And then seeing that they didn't see the screenshots. But I imagine if you zoom out and you have multiple data points altogether, it's more useful. Like, do do you have like, I don't I I do not want to say a top three or top five because, you know, three, four, five, like, doesn't matter, but, like, what are essential data points that you would need to come up with, let's say, a useful recommendation for how to improve a product?
Tyler:Good question. So when you say data points, meaning just like if I'm benchmarking any feature, what I'm a look what I'm
Nick:Yes. So yeah. So meaning that the recording of a user going through your prototype perhaps is one. The thing the questionnaire in the end is another one. And then I'm also thinking about, you know, post hall or or hot jar or, you know, so so because I was a bit stuck.
Nick:You know? But the question the answers I got were very eye opening and and silly and sad in in in in their own way. So I'm basically looking for a you know, if we're going to do this again, like, what's a better way to set this up and make
Tyler:it feel more useful? Okay. So my bare minimum or, like, what I love to do, like, this doesn't always happen, but I think it would just like a funnel. And I I do it in three separate steps. So I have the discovery, which will be a I like to do an in app survey.
Tyler:So, like, you can trigger a modal. You can actually get pretty fancy with this. You can go, hey. Or get some engagements. Go, hey.
Tyler:We're thinking of launching these five features. Which one of them would you love to have first? And then you're able to kind of get some information. You're focused on the one, but you're hopefully, you're getting some traction there. And then the sec follow-up is, if we launch this feature, what are the top five things that are important to you?
Tyler:And that's like your first net. Okay. Cool. We have and that you can pair with your peer PRD, but you're like, here here are the top three things that our users want to be included in this feature. Cool.
Tyler:Now I have my preliminary discovery work. I'm working on this feature, and now I have a prototype or like a high fidelity thing that I wanna test. And then I'll run a usability study that go and then my my KPI there is just completion rate percentage or time, whatever you wanna do it. So, like, everyone was able to 60% of people were able to complete the task or people were able to complete it under a minute. And then you can do your your feedback cycle there.
Tyler:And then the last thing is when you launch it, do they use it? So that's the adoption rate. Right. So I think those three touch points, I know that's a lot of research, but I think, ideally, that's what you want. It's like, here's the here's what they want.
Tyler:Let's test if it's useful. Can they complete it? Or you can follow-up with a questionnaire. And then are they using it? That's the distinction that we mentioned early earlier is users say what they say one thing, but they do another.
Tyler:So let's see if they actually use it.
Nick:Right. So so there are questions in the beginning. Mhmm. Discovery phase. Yeah.
Nick:Do you really ask anything or talk to people in the later stages?
Tyler:I ideally, I like to carry the same users. So, like, if I send out that survey and they ask Mhmm. They answer my questions, I'll pull them in to the the meet the the middle phase, which is, like, the the usability studies. Like, hey. I heard that you had some feedback.
Tyler:We're building it now. Would you like to see it first before it actually launches? That'll get them in, and then I'll test it. And then that's the information I get. And then I'll I'll do the analytics hot jar, see if my adoption rate is up.
Tyler:Mhmm. But as an additional sanity check, I'll say, hey, we've launched it. What do you think? Like, we hit or missed the mark. Like, you've covered your your bases.
Tyler:And then in that ideal world where you're you're working in iterations, I now I now know based on that feedback at the end what the next feature or next, like, the v two should be.
Nick:Right. Right. Right. Yeah. That makes sense.
Nick:Yeah. So it's it's it's not really about that question I asked in the beginning, but it's more like that the fact that it was isolated. Like, it was the only thing we did. That it's a bit strange or useless, quote unquote. Yeah.
Nick:Alright. Well, that's helpful, you know, because we were going to end up just watching recordings and not doing anything else.
Tyler:Yeah. It's really just depends on what information you wanna gather. And and Mhmm. I always advocate for more the better or at least one one data point is good.
Nick:Well, that's always better than zero because it it's been useful. You know, just realizing that people can just say something is a good thing to know because we will be better because of it in our approach to testing. Yeah. It's it's it's it's
Tyler:it's a learning every experiment or test is something you learn regardless of how well it learns. Over time you learn you learn how to talk to people and get them get the information.
Nick:Mhmm.
Tyler:Be the salesman the other way. Just make sure you're getting the the juicy information that you want.
Nick:I feel like it's a a a craft, a skill, a research, user research that's it's it's getting more to the to the background. Perhaps it's an AI thing. Perhaps it's because it's hard to sell. It's much harder to sell compared to a shiny visual of a dashboard. Mhmm.
Nick:But I yeah. I'm somewhat concerned. I'm not sure if that's valid or not, but I feel like we're doing less research and not and and the research we are doing is perhaps of a lower quality in general. Not saying that there aren't any pros there, but Yeah. I'll feel like we're doing less.
Tyler:I'll say that there is and there's gonna be increasing value over time. The people that are smart enough to kind of take up the take up the charge, research is gonna be very valuable. I'm seeing a lot of, like, the development cycle change. So it's development is becoming faster and cheaper to do. You can get more things out of the door.
Tyler:And at the end the day, the the the prompt you create dictates the solid output. I think there is more value in the input, which is the research part. So how that might look like practically is you're able to kind of vibe code an experience.
Nick:But
Tyler:if you if your source material or your input is a slew of data points in terms of user research, conversations, ranking, your your score, like, all that information aggregated together as, like, the input for the thing you're creating, that's, like, undeniable.
Nick:Yeah. Yeah. And that makes sense. I feel relieved, actually. And your research knowledge and expertise really really set me straight.
Tyler:Good. I'm glad. Research is important. The answer to the answer the looming question, like, yeah. You can you can you can like, this like, in 2011, we used to not do any research, and that didn't work because, like Mhmm.
Tyler:It was the I was always the CEO. I know what to build. But really Yeah. Really, you don't. You you find your avatar.
Tyler:They know they they like it or they don't like it, and then you keep building for them.
Nick:That's a big red flag. Right? Designing something because of a CEO gut feeling.
Tyler:Yeah. Unless he's unless that CEO is talking to their customers on a daily basis. I've seen that. Yeah. So, like, I've seen the good version of that.
Tyler:It's like, I I know the users and the they're they have the paper trail to back it up. Like, they're on sales conversations. They're talking to the end consumer. They're they're they're on the pulse. Cool.
Tyler:But at some point, your your your job as the CEO is not is to do other things. Like, that's why you hire, like, designers and stuff.
Nick:Yeah. True. Well, I I think it's a good idea for any CEO to, once a week, call customer support, You know? Not telling them you're the CEO. Just just telling, like like, you know, just as a as a random person.
Nick:Hello. I'm John, and I'm stuck. Can you help me? And just to see how quickly they pick up the phone, how friendly they are, how useful it all is. You know, just to yeah.
Nick:It's like that old show, that undercover boss thing. I think it's what it's called, you know, where you where the CEO becomes a janitor for a week in his own company. I think that's useful. Maybe we should do that as designers as well. Yeah.
Nick:Side note, let's not let's not let's not go off track here
Tyler:too much. Bring me back to Undercover Boss, solid show. Yeah. But I dig Yeah. Digress.
Nick:Yeah. Yeah. Yeah. Yeah. Well, I I certainly feel helped, so thank you again.
Nick:And, yeah, I don't have any closing thoughts here. I'm just happy now.
Tyler:Good. I think you're end up saying research is very important.
Nick:Yes.
Tyler:The I think there should be attempt to do a two research message methods per project. I think it's a really the ROI is is there. It's just how you pitch it. Yeah. True.
Nick:Well, on that note, let's wrap up for today. Alright. Till next time. Till next time.
Tyler:That was a great episode. So if you like this content and wanna hear more, please like and subscribe.
Nick:Yeah. And if you want to see more, please go to designtablepodcast.com, Spotify, Apple Music, all the big players, and more.