How I Tested That

In this episode we interview Hala Saleh. Hala is a skilled and thoughtful product leader who I met by coincidence over 10 years ago while reading a copy of the Lean Startup book. 

We catch up and discuss how the term Minimum Viable Product or MVP has evolved since then and how lean startup principles can be applied outside of the digital product space to things like communities.

Hala shares her journey through various roles in tech over the years while emphasizing the need for testing and customer feedback.

I really appreciate her thought process on how she tackles problems in different industries.

Is your innovation pipeline clogged?
  •  Uncover the risks, bottlenecks, and gaps holding your best ideas back.
  •  With the EMT Diagnostic, you'll get a clear, actionable plan to fix them.
πŸ‘‰ Book a free discovery call at https://www.precoil.com/innovation-diagnostic

What is How I Tested That?

Testing your ideas against reality can be challenging. Not everything will go as planned. It’s about keeping an open mind, having a clear hypothesis and running multiple tests to see if you have enough directional evidence to keep going.

This is the How I Tested That Podcast, where David J Bland connects with entrepreneurs and innovators who had the courage to test their ideas with real people, in the market, with sometimes surprising results.

Join us as we explore the ups and downs of experimentation… together.

David J Bland (0:1.346)

Welcome to the podcast. Hello.

Hala Saleh (0:3.618)

Hey, thanks for having me, David.

David J Bland (0:6.050)

I'm so excited to have you on. I have to start this podcast though, with how I first met you, which I think was on an airplane. ⁓ I think it was ⁓ 2012 maybe or 11. I don't remember the specific year, ⁓ but we were going to a conference and we just ended up on a plane together ⁓ and you were reading Lean Startup Book. And I was like, I know that book. ⁓ And then we started chatting about testing and everything.

Hala Saleh (0:12.452)

⁓ one. ⁓

Hala Saleh (0:22.436)

Mm-hmm.

Hala Saleh (0:28.995)

Yeah.

David J Bland (0:36.425)

And I thought, wow, I need to really get you back on here to talk about what you've been up to since then. Because you were sort of, in my view anyway, sort of an early adopter of that approach with MVPs and everything. But I would love for you to share a little bit about yourself before we jump into what you've been up to since then.

Hala Saleh (0:52.482)

Yeah, I'm so glad that we reconnected. ⁓ It's funny to think that it's been that many years ⁓ and a little bit wild ⁓ to think that we've been doing this like MVP and testing and ⁓ tech thing for this long. ⁓ so ⁓ yeah, my name's Hala. I've been... ⁓

kind of in a sense as I was thinking about this episode, ⁓ I realized that I've just kind of been ⁓ doing MVP-ing, ⁓ right? And we'll talk about MVP and how the evolution of that ⁓ term has kind of evolved or how people think about it now. ⁓ But just like I've been testing things. ⁓

for my whole career. Like, I've been testing my career for my whole career. you know, ⁓ do I want to do this thing? Do I want to do that thing? I started out as a developer. Then I was like, hey, maybe I want to try project management because we always end up having issues with our delivery. It never feels like it's on time. We're delivering the things that the stakeholders want. Okay, so I'll try project management. And then I was like, there's this new thing in project management called Agile. ⁓ And like, maybe I want to test doing that for a little bit and along the way. ⁓

⁓ Along the way, convincing people to let me do these new things that I wanted to test, whether they were my managers, ⁓ people who I wanted to be my managers, CEOs of companies where I'm like, ⁓ hey, ⁓ I know you think that you want me to be a project manager in your company, but really what you need is a director of product. So ⁓ I'm your woman. ⁓ So it's just been like very much a theme in my career and then in how I do things. ⁓

⁓ And I can give more examples as we go through and chat about that. ⁓ But really like this realization came as I was thinking about talking to you about this episode because initially I was like, I'll talk about testing methodologies and the different ways that we test things ⁓ in the world. And then I was like, no, actually, it's a much bigger theme for me and that's just kind of been my path. ⁓ So yeah.

David J Bland (2:58.848)

I love that. I think ⁓ I have a hard time with uncertainty, ⁓ which might sound weird coming from somebody that wrote a book on how to test the business side is. ⁓ And I live in that world, but I find that ⁓ it's helpful for me to have, you know, like a process of dealing with it. ⁓ And ⁓ interesting enough, like ⁓ I got pulled into agile pretty early in my career. Like the first startup I joined, like you, I've worn many hats over the years and the first startup I joined.

Hala Saleh (3:3.044)

Yeah.

Hala Saleh (3:6.968)

Yeah. ⁓

David J Bland (3:26.114)

I was asked to do so many different things. at the same time, we were like, now we're agile. And we didn't really have any training or anything. We were just trying to do these stories and sprints and all that and everything. But I realized ⁓ a lot of the agile community is also risk averse, which I didn't think so at first. I thought, these are going be a bunch of risk takers. And they want to do this thing called agile and everything. And what I realized, it was a lot of people that had a hard time with uncertainty. And they needed a process to go through it. And I think over the years, ⁓

Hala Saleh (3:47.277)

my God.

David J Bland (3:55.426)

that's been reinforced through the people I've met through that community is similar is, like, this gives me a way to deal with a lot of that uncertainty being able to iterate through, you know, a potential solution.

Hala Saleh (4:5.728)

Yeah, mean, the whole one of the whole points is to create predictability, right? ⁓ Is that, you know, we end up ⁓ planning these big projects and doing big upfront planning ⁓ and then it all. ⁓

goes to heck. I don't know about how you feel about cursing on your show. and so let's create this process where we have check-ins every couple of weeks or whatever. And we're see, you how things are going so that we have predictability. And, and then came in like all the metrics and then came in all of the measuring and then came in the velocity and then came in the being religious about velocity and then came in the scaled agile and the, the, you know, measure 500 times.

⁓ cut 500 times but ⁓ it just kind of got wild after there and and so I agree I think that a lot of people who were in agile were

kind of risk averse. And that's where I think ⁓ the beauty of having the lean startup approaches and just kind of the testing things out, ⁓ taking some of the concepts from Agile and then mixing that with the entrepreneurial spirit has been really interesting to kind of find the right ⁓ recipe around that, I guess.

David J Bland (5:21.442)

Yeah, I agree. think with Agile, it's really great at iteratively delivering solutions. ⁓ And then at its core, you should also be talking to the customer. Obviously, you go back to old XP literature and everything from Kent Beck, and they sort of like put the customer right next to the developer, you know? And we've kind of prided ourselves in getting many layers in between them ever since. ⁓ But, ⁓ you know, when I think about the stuff that you and I are drawn to, it's often, well,

Hala Saleh (5:36.718)

Yeah.

David J Bland (5:47.788)

Are we solving a problem? Do we understand the problem enough to even take that approach? Because you can iterate through solutions to something you don't fully understand and it can take you all kinds of different directions and you could still fail in a kind of a big way. ⁓ And so I do think applying that methodology or that just framework of ⁓ how to be undercover the problem. And I think, you know, just through your intro ⁓ and sharing kind of ⁓ you would spot opportunities to say, well, I think you need me for this, but then it looks like

this is the real issue going on. I find that helps build empathy as well, ⁓ playing all these different roles. When someone comes to you and say, well, you don't know what it's like to be a project manager, say, well, yeah, I kind of do. ⁓ I've tried that. ⁓ It's like early on in my career, I played so many different hats. I was doing testing for a while, traditional QA testing. And people were like, you don't want to be a tester in Agile. I was like, actually, yeah, I know. It's not real fun when you get to the end of the sprint and everyone

Hala Saleh (6:30.660)

Thank

David J Bland (6:46.126)

closes all the stories at once and hands them over to you. I know what that feels like. So I do think it gives you some empathy or a way to relate to people in ways where they just assume that maybe, oh, you don't understand my world.

Hala Saleh (6:59.496)

100%, I mean, I was a QA tester too. I did ⁓ what we called at the time, like white box testing, black box testing. I ⁓ had to write scripts, I had to write automated tests. ⁓ Again, ⁓ it helped me be able to speak the language of the teams that I work with moving forward from there. ⁓ And ⁓ it ⁓ actually... ⁓

I think can be a superpower for people who have worn multiple hats to then be able to work on teams because you can actually really ⁓ have the teams back in critical moments where the stakeholders are pushing and you have ⁓ a position of authority or influence or whatever on a team ⁓ to be able to speak to ⁓ why things are the way they are or understand obviously also gives you leverage to challenge the teams because you're like

Yo, I've done that too, you know, ⁓ I understand what's going on. ⁓ But it helps in both ways. ⁓ you know, going back to your point about understanding the problem and the problem space, that is like my food. That is like, I don't think that people really appreciate... ⁓

sometimes how important it is not only to really steep yourself in understanding the problem and the problem space, but staying ⁓ in it, continuously testing whether the problem space has changed, ⁓ continuously trying to understand how the problem space has changed and evolved, because nothing is static. The problem space isn't static. The humans who are experiencing the problem aren't static. The tools that they have available to them, the things, the way that life changes,

and the world changes ⁓ isn't static. And so just thinking that I understand the problem now, so now I can come up with a solution and that solution is gonna ⁓ work for the next ⁓ 20 years or whatever because I've figured it out ⁓ is a killer for a lot of us in our jobs and in our roles. ⁓ And that's the thinking around continuous discovery, continuous discovery, continuous discovery comes in, right? And some of the work that some people have been doing such great work

Hala Saleh (9:13.638)

out there in the product space like Teresa Torres, right? And continuous discovery and always checking and always testing is just so important and it's where I feel like for me I'm always trying to understand the problem and the humans. The problem and the humans.

And it's funny that, I mean, just again, also coming back to the point on the early literature ⁓ and whether it's Agile or any of that stuff, ⁓ I think I realized also ⁓ in the spirit of testing that I learn ⁓ the best when I actually try something out. so ⁓ back in, goodness, I'm gonna now really age myself, ⁓ 2000. ⁓

I was interning at a software company ⁓ and ⁓ I ⁓ told the team that I was working with, the developers, I was like, I am going to deliver a brown bag session for you and it's going to be on this thing called XB. ⁓ And ⁓ like I didn't really know a lot about... ⁓

David J Bland (10:18.227)

amazing.

Hala Saleh (10:24.226)

what it was, I mean I was super still new to even coding, ⁓ but I was like, we're gonna talk about this thing. And ⁓ I made a presentation and I talked to the team about ⁓ extreme programming and I like ⁓ went, and so even moving forward in my career, every time, like when the Lean Startup stuff was really... ⁓

becoming popular. That's when I was reading the book. And then I went back and ⁓ lean canvas or whatever canvas is I would come back and, and when I learned something, I turned it into a workshop and I taught other people. Now this doesn't mean that at that time that I was giving the workshops, I was an expert at these things, but it like actually helped me understand ⁓ not only the thing, but also ⁓ what is people's reaction to it? Where are the problem points with it and the friction points? And it, and it allowed me then to, ⁓

like take the things that I thought were working well within these frameworks ⁓ and drop the things that ⁓ I felt were just like either... ⁓

you know, not needed, they were too much. ⁓ It actually didn't contribute value to the framework or whatever. And that includes things like Agile. ⁓ So I think it's just like that mentality of continuously testing and checking ⁓ and taking something out into the real world and trying it out is the best way versus just looking at things in theory or trying to apply things, ⁓ you know, by the book. ⁓

David J Bland (11:48.610)

Yeah, I think quite often these books are written in a way that you almost have to make some compromises as you're working in the ⁓ print. ⁓ It's got to start and end and you have to lay it out in a way. ⁓ But it really limits you into ⁓ how you mix things up, how you be iterative. You ⁓ maybe you don't start with this tool. You start with this other one instead based on your context. Or maybe you don't even use this other tool based on where you're at. ⁓ And I do think ⁓

learning by doing, know, and obviously you're being really thoughtful about how you're applying it and everything, but I think that's how it sticks. I mean, we can only learn so much through case studies and all that. And that's one the reasons I like taking companies through their real ideas when I apply my framework, because I was like, yeah, I can teach case studies all day long, but the only way this is really gonna find out if it works for you or not is by doing it on an idea that you have. And so I like that approach. And something you said about staying in the problem space I thought was very insightful.

What do you do to incentivize or help people stay in that space? Because ⁓ I ⁓ think see teams, ⁓ either they feel uncomfortable there, or they feel awkward, or they feel as if I want to just work on the solution now. ⁓ How do you approach that?

Hala Saleh (13:4.450)

Yeah, ⁓ it's a really good question. ⁓ I think, ⁓ so I ⁓ don't want to lose some of the, I have a number of thoughts around this. Number one, it's important to just drive the point home that it ⁓ is really important to understand the problem before starting with any solutions. ⁓

I actually feel like even when I was doing program and project management years and years ago, this is like 2007, 2008 at Hewlett Packard, right? One of the first things that we learned even as just doing program management across different departments was the importance of understanding the customer's experience. they had us as teams go and do customer visits and sit in the customer's home and watch them unbox the printer and set it

⁓ up ⁓ and go through installing it and ⁓ it's a humbling experience because you can't be like, ⁓ give me your keyboard, I'll help you set this thing up. ⁓ It's so easy because we developed ⁓ the process and the onboarding experience and you just need to do it this way. You have to watch the customer go through it and that was echoed through different experiences I had even at Expedia. So we'd have people come into a lab and put the software in

of them and we just had to watch. ⁓ And so it's very humbling when you watch people use things that you built ⁓ and you think, it's so intuitive and so easy and finding the real pain points. ⁓ So I think that building in some level of customer interaction for different ⁓ teams, even if it doesn't feel like they need to be customer facing teams, ⁓ is super important. ⁓

anybody can do it, right? You don't have to be ⁓ a manager or leader or whatever to ⁓ incorporate that into your team's ⁓ workflow. ⁓ You can have influence and influence people to incorporate it and say like, I think this is really important and here's why. ⁓ But it doesn't have to be complicated. It can be as easy as having people that you know ⁓ use the software and ⁓ figuring things out from watching how they use it or what issues they have.

Hala Saleh (15:21.574)

⁓ Nowadays, we have access to social media, going on social media and seeing what are people saying about your product. ⁓ But I think as somebody in my role, what I try to do is, ⁓ when I ⁓ joined that startup that I told you, I convinced them, you need a director of product, so hire me for that. I had the developers all rotate getting on customer support calls. ⁓ They had to answer the phone. ⁓

⁓ and talk to the customers about their issues and their problems. And it was funny because initially I thought it would just drive a sense of empathy and understanding the problem more. ⁓ And sometimes it ended up being that the engineering team would be like, next sprint, all we're doing is solving these problems that we heard about on these calls this week because we are so embarrassed about ⁓ whatever that we heard and we just want to fix it and we want to make it better for customers. Like the empathy went through the roof.

where before it was just like you're behind a screen and you're doing your engineering thing and ⁓ excited about the code you're writing, ⁓ but by actually having interactions with customers, like, ⁓ there's really a set of problems that we need to solve. ⁓ So having ⁓ kind of forcing. ⁓

or incorporating is the nicer word, ⁓ interactions between teams and customers ⁓ and being creative about what those interactions look like. The last company that I was at, we had a Discord ⁓ server for ⁓ the community that was involved in the product or had interacted with the product or had actually invested through the platform or whatever. ⁓ And so just going in and listening or watching some of those conversations, even if you're not interacting. ⁓

But reading some of those conversations, incorporating a certain amount of time, percentage of time where you're expected to go every week and like either read or interact or answer questions. Those are some of the things. ⁓ But also as a product person, making sure that on a cyclical and regular basis, ⁓ bringing people back to the problem space by incorporating new data and new learnings. Learnings, ⁓ right? And having a sense of like learning is our number one job.

Hala Saleh (17:37.750)

⁓ as we're putting a product out and measuring and learning ⁓ and bringing those learnings back, ⁓ whether it's on a monthly basis or whatever, here's how our product is doing that we thought that we released to production and now it's ⁓ just humming along. ⁓ Here's some data points from customers that we're getting, whether it's from Discord, whether it's from a survey, whether it's from ⁓ customer interviews, whether it's me picking up the ⁓ phone and calling customers who have canceled.

transactions or gone to something and canceled and understanding why and having assumptions about because they don't like our product and learning that it's because they were scared or they didn't have enough information or there was something that they just needed a little more handholding and can we then incorporate that into the product? So it's just finding ways to continuously be interacting with the customer that is using the thing that we're building.

David J Bland (18:35.886)

I love that. I think it's a very important point to make that we stay in that problem space and we create opportunities for our team members and our peers to explore that problem space. think that's one of the things that drew me into this line of work was with Agile, felt as if, I got Agile. I like the idea of iterating through potential solutions, but I think...

you know, not knowing the problem well enough and seeing the churn through what I thought were highly functioning agile teams, but ultimately delivered something that people didn't want. It really drew me into this kind of lean startup thing and all that, which is how we originally met. I have to come back to that for a moment because it's been a while. It's kind of crazy to think how long it's been since that. think it was 2012, if I remember correctly, or 11. 11 was when the book was released.

And what are your thoughts on it now, you know, taking what you've tried and applying it over the years, you know, a lot has changed in the market and technology now and how we even view customers. What was your take on on that book and some of the methods and some of the ideas like MVP that popularized?

Hala Saleh (19:52.704)

Yeah, ⁓ I think that, ⁓ I think it is still a very valuable ⁓ mindset and paradigm for people to learn and understand. ⁓ I think that ⁓ it ⁓ is still also very challenging for people who are in executive roles and or ⁓

running enterprise level companies and platforms ⁓ to be willing to think in that paradigm. And so the tension that existed then, I think in some ways still exists now. ⁓ I think that it's up to... ⁓

people who are in a position to translate the paradigm and the ideas from Lean Startup into how do I apply this in different environments? And that is a skill set on its own because you can't do some of the things that you can do in a startup, a full startup. Like I taught a class at University of Washington for a couple of quarters a couple of years ago or about a year and a half ago on product strategy.

And one of the classes was testing methods ⁓ and ⁓ how we test things, ⁓ our software and whatnot. And so people were like, do you really think we ⁓ should use the, ⁓ what's that method called? The smoke and mirrors, ⁓ where you kind of just have a ⁓ UI, but it doesn't really do anything in the backend and it's to test something.

David J Bland (21:27.126)

All sort of like Wizard of Oz type thing or fake door. Yeah.

Hala Saleh (21:30.939)

So, do you really think that we should be, or how do I use Wizard of Oz testing? They got really excited about that and I'm like, hold up. That was great in 2011. You could get away with a lot.

But, and you still can, but ⁓ it needs to evolve to where people's expectations of technology are these days, right? What people's expectations of ⁓ service, even customer service are these days. And that has all changed over the last, you know, 14, whatever years, 13, 14 years. ⁓ And so ⁓ the methods and the thinking and the paradigm is very important. How can you... ⁓

learn the most ⁓ with the least investment. ⁓ And I've taken that and actually applied it to things outside of software. ⁓ Now, ⁓

With that being said, when it is software, we have to really be careful about using methods that then, especially if you're at an enterprise level and you have products in ⁓ production and people are using them and you have a business and you have a business model and you have profits and loss ⁓ and all of that, you can't put up products that ⁓ are wonky or don't actually work or make a promise that you don't deliver on. ⁓ And so ⁓ those things are really important to understand context. So the paradigms and the

methods are important and are good, but then apply context to them appropriately. ⁓ Apply ⁓ the landscape to them appropriately. ⁓ Can you use some of the ⁓ methods ⁓ while you're in just the prototyping phase, in the user testing phase, in the user research phase? And obviously we have a lot of evolution of tools now and AI tools and all of that for prototyping and such, ⁓ making sure that you're using the methods ⁓ that are in that book.

Hala Saleh (23:27.034)

that are developed ⁓ appropriately in the right context. ⁓ So that's what I'll say. But yeah, ⁓ think it's still super valuable. ⁓ again, I've kind of taken that and applied it to even the physical world ⁓ and through some of the projects ⁓ that I've been working on. ⁓ And it's really served me well.

David J Bland (23:49.634)

Yeah, I think ⁓ the context is really important, especially usually the companies I'm working with, they don't want to damage their brand. You know, it's like my brand's over our brand's over 100 years old, you know, we can't ⁓ can't damage the brand. But then it's a balance between well, we don't want to damage the brand. also don't want to perfectly deliver something polished that doesn't have a need. ⁓ So where's the ⁓ where's the ⁓ trade off there? Where's the compromise What about brick and mortar? Have you seen where you can apply this thinking or have you seen stuff that works?

Hala Saleh (27:9.476)

⁓ Well, I mean, I can speak to ⁓ only like what I've tried and most recently I kind of came at it also from the problem space, so that's where I started ⁓ was I was experiencing ⁓ an issue where I felt like I

there was all this talk about community, community, community. And then when it came down to it at the end of the day, I was like, ⁓ who do I feel like I can call if I'm in need ⁓ or a group of people that I feel like can support me if I need something, need help? ⁓

and need community support. ⁓ And I didn't really feel like I was at a place where ⁓ I felt like if I fell there was a community to catch me. ⁓ It felt like we were doing that online a lot, but ⁓ it just like ⁓ there was a disconnect between the digital world and the word community and how it was being used in digital spaces and what I felt like was lacking in the physical world. ⁓ And I have had a feeling over the last few years ⁓ and this kind of

hypothesis that ⁓ the speed of technological advancement, and you you and I have actually experienced it over the last four or whatever decades, ⁓ and we've been through the phases of the desktop IBMs and whatever through to...

you know, AI. ⁓ And so, but the speed of technological advancement has obviously been ⁓ quite high over the last few decades. but I feel like we're actually at a cusp right now ⁓ where it's, ⁓ if it hasn't already, it's getting to a point where it's going to accelerate at such a rate ⁓ that as humans, cannot adapt to it. Like we're going to be stuck in a spot where we're like, we don't even know what to do with this. ⁓ And, ⁓

Hala Saleh (29:5.090)

And I feel like the only way to build resilience ⁓ with that happening, it's not to close our eyes and pretend it's not happening, ⁓ but rather it's to ⁓ create a resiliency by knowing how to live in the physical world ⁓ and making sure that we're practicing living in the physical world. ⁓ And so with that and just ⁓ the sense that ⁓ there was this idea of community that I felt needed to actually be built one-on-one.

I wanted to test out the idea of building a community space. We've all heard about third places and ⁓ all of that. ⁓ I have a very specific community that I'm from that has very specific needs right now with regards to feeling safe and ⁓ the togetherness and ⁓ reinforcement of identity and of our culture and all of that. ⁓

I'm Palestinian and it feels like an identity that is kind of at risk right now in the world. And so I wanted to test the idea of the need, number one, is there demand, right? Is there a need for people to come together in community spaces and does it...

⁓ check off the boxes that I think it would need to check off for it to be considered successful. Like what are my success metrics? And so I created kind of I documented what the problem was, ⁓ who the person is that's experiencing those problems, what the ideal solution looks like, ⁓ or the vision, right? The North Star and the vision. ⁓ And then I realized that ⁓ I needed to test this out. I have myself as a data point, which we all know. ⁓

Sample size of one and that one is me is enough ⁓ data, right? ⁓ So ⁓ I found a way to do this through a pop-up. And I realized the pop-up was my MVP. ⁓ The pop-up ended up lasting for nine weeks, which was amazing. ⁓

Hala Saleh (31:8.644)

very lucky that I was able to test ⁓ for that long, but it was an actual physical test of the idea and the hypothesis that I had. ⁓ And I did everything you would do with a product. I ⁓ created what are my success metrics, what are the KPIs ⁓ that tell me whether this is successful or not. ⁓ We collected learnings, ⁓ measured, ⁓ customer feedback, customer interviews, all of that in order to make sure that this

actually a thing that if it were created in the real world in a more permanent sense, ⁓ you know, what's the criteria for an MVP so that it actually can be a proxy for the real product at the end of the day versus like if I create an MVP that is so way off base of what the end product is going to be, ⁓ then what am I really learning?

⁓ So that was the mindset that I approached this and created that pop-up space.

David J Bland (32:9.262)

Interesting. I like that having a physical space ⁓ that's not necessarily just, ⁓ we're going to meet over here. ⁓ It seems like you were trying to create a sense of belonging to here's something that we can connect into, and this is a safe space. I'm wondering, ⁓ I love that you took that process of, I'm going to define all my criteria. ⁓ mean, from whatever you can share, what kind of criteria would you use to?

Determine success for that because I because I find myself if I had to come up with it I might struggle with how do I know I mean granted I could say how likely are you to recommend this community? ⁓ But like beyond that I like I think I would struggle coming up with what I measure there

Hala Saleh (32:56.098)

Yeah, ⁓ I mean, number one, we measured attendance. So vanity metric of like, right, like page visits, right? But it wasn't page visits. It was actual like ⁓ cafe visits. Like how, ⁓ well, ⁓ what, how many people were coming through the space ⁓ on a daily basis when it was open? ⁓ And ⁓

⁓ The next metric was actually repeat customers. So were people coming back? And at what rate were people coming back? ⁓ And then there was also virality. So when people were coming back, were they bringing somebody else with them? How many people on average were people bringing with them? ⁓ That was one metric, but virality also was how many people were coming in? ⁓ Sorry. ⁓

referrals. So how many people were coming in? Because they were referred by somebody or they heard about it from somebody and so they came in and and that's the the driver that brought them in through the door. ⁓

And so those were some of the metrics that we looked at and that I looked at when I was measuring whether this thing was successful. And ⁓ some of that was ⁓ able to be tracked ⁓ quantitatively. Like I could see ⁓ orders and tracking and whatnot. But also I had events. ⁓

twice a week we would have a ⁓ community events and they were either educational or cultural or whatever ⁓ and ⁓ technically like I didn't actually need people to RSVP to these events but I created an RSVP process because then I could track people's emails and then I could track people's ⁓ attendance and I could track if they came back for events and how many people who were at the next event had been at a previous event. What was a percentage of that? How many people were new customers? ⁓

Hala Saleh (34:45.830)

⁓ and tracking ⁓ all of that as well. ⁓ So those were ways that we were tracking, in addition to just getting customer feedback through testimonials or... ⁓

⁓ a survey that we had available to people and talk about their experience and whether they would recommend it to others and bring others. And then a very qualitative one was ⁓ when I had the last weekend of it and it was like, we're taking a break for the month of June and we're gonna figure out our long-term plans. ⁓ And on that last day, ⁓ I'm walking around and chatting with people and people are pulling me to sit with them at their table. Okay, what do we need to do to make this thing a reality, to keep it going? ⁓

So that was a very qualitative metric, but in addition to the other ⁓ ones.

David J Bland (35:32.940)

Yeah, I like how you're balancing both qualitative and quantitative there. And I feel like that could be challenging with something like a community. ⁓ And I do think ⁓ it reminds me in a weird way of something like a Sean Ellis test where it's, you know, how disappointing would you be if this went away now that you've experienced it? I don't think you manufacture that you're like, look, we have to take a break before we figure out what to do next. And to have people go, no, I would be very disappointed if this went away. How can I, you know, contribute? I think that is ⁓

It's interesting thing to ⁓ receive feedback on as well. interesting. So you've kind of taken what you've learned over the decades of product management and testing and applied it to ⁓ almost like a community or a physical space. ⁓ What do you think, if you're willing to share, what do you think the big assumptions are that you want to test sort of now with us bringing it forward? Because I think it sounds as if maybe you've

You have some directional evidence that some of this is trending ⁓ in a way where you can persevere. But what are your big assumptions now with it?

Hala Saleh (36:38.820)

⁓ So the big assumptions, ⁓ I mean...

The MVP, while it had a number of metrics that I would say checked off as successful, ⁓ one of the things that wasn't a priority during the MVP phase was profitability. ⁓ And so ⁓ the financials are going to be a big assumption moving forward on whether it can actually sustain itself ⁓ and whether it can be something that is financially viable. ⁓ And so that is something that ⁓ if ⁓ we are able to have

⁓ of...

a different iteration of ⁓ the MVP, ⁓ potentially where the profitability aspect can be tested ⁓ is going to be important. ⁓ Community investment, ⁓ not just by coming and experiencing the space, but are people willing to actually invest ⁓ their dollars in making this a reality is something that we can also test actually through ⁓ putting up something like a Kickstarter.

⁓ And seeing what does that look like? What are the results from that? How many people are willing to ⁓ actually put their money where they're saying that they want this to exist in the world? So are you willing to also be a part of its success? ⁓ And then actually like longevity ⁓ of it. ⁓ Is the need a long-term need? ⁓ Is it, you know, is it... ⁓

Hala Saleh (38:13.442)

You never want to be solving a problem for something that's just a trend ⁓ or something that's just ⁓ a temporary need, but rather is it a long-term need? And I think ⁓ that some of that ⁓ has to do with looking back in history ⁓ and seeing how long has this need existed. I don't think that it's ⁓ a temporary need, but it's just kind of testing whether ⁓ the way that we are delivering ⁓ the community.

The way we're delivering to the community also needs to evolve as the community evolves. So in the spirit of what we talked about earlier of staying in the problem space, what the community needs today is not what they needed two years ago, is not what they're going to need six months from now.

David J Bland (39:3.534)

I like that. I like staying in the problem space with the community. And I think anyone listening to this, trying to build a community, you know, that's good advice of staying in the problem space and understanding how their needs change. ⁓ I just want to thank you so much for hanging out with me and chatting about what you've been up to since 2012. ⁓ Everything from, you know, all the different roles you've played in different companies, how you've applied, you know, things that you've read in

you know, through brown bags, but also just applying them with teams and little ways to see how they respond. I love that. All the way up to how do we do this to build out a community? I just really appreciate you sharing your journey. If people have listened to this and they want to reach out to you and are like, I need to ask her for help. ⁓ What's the best way for them to get in touch with you?

Hala Saleh (39:51.620)

⁓ Probably at this point LinkedIn. ⁓ I ⁓ am working on trying to relaunch my website but it's just it's not quite there yet so when it is you'll be the first to know and then we can share it out. ⁓ For now probably LinkedIn is the best way.

David J Bland (40:9.644)

Alright, so we will make it a point to have ⁓ LinkedIn on the detail page. So if you all want to reach out through that as well. Thank you so much for hanging out with us and sharing your stories and I just appreciate you spending time with.

Hala Saleh (40:21.326)

Thank you, it was so fun.