Subscribe
Share
Share
Embed
IIIMPACT is a Product UX Design and Development Strategy Consulting Agency.
We emphasize strategic planning, intuitive UX design, and better collaboration between business, design to development. By integrating best practices with our clients, we not only speed up market entry but also enhance the overall quality of software products. We help our clients launch better products, faster.
We explore topics about product, strategy, design and development. Hear stories and learnings on how our experienced team has helped launch 100s of software products in almost every industry vertical.
MIT report revealed that 90% of companies investing in AI tools like ChatGPT, you know, they're not seeing any ROI. Only about 5% of AI investments yield real results.
Speaker 2:The speed at which technological innovation occurs is so far beyond our ability to keep up with it that there's no way that you can stay competent.
Speaker 1:Alright, everybody. Welcome back to another episode of make an impact podcast. I'm your host, Makoto Kern. I'm here with Brinley Evans.
Speaker 2:Oh, this is where I was supposed the part that I'm supposed to say. Hey, everyone.
Speaker 1:It's alright. It's good. This we we've done tons of these. But now we've got a I think we've got a pretty exciting topic today and it's something that I've coined the term, the AI competence mirage. And we're pretty much trying to pull back that curtain on whether smart leaders think they get AI, but we really think they don't.
Speaker 1:And so recently a real estate exec that I know of, they talked about their AI tool and they brought up a really interesting point when they brought it up to other executives. They just ask basic questions and they're looking at AI as like a chat GPT, but it's more like a chatbot tool. And, you know, it's something that could really easily be done for free and completed in just thirty seconds. Also, think another big stat that we've been seeing or we read recently was a MIT report revealed that 90% of companies investing in AI tools like ChatGPT, they're not seeing any ROI. Only about 5% of AI investments yield real results.
Speaker 2:It's really not the technology at fault. Lots of companies fail because AI just doesn't stick, isn't able to adapt, can't retain context. So it's pretty much sidelined after being seen as a sort of flashy demo or, you know, really just not an actual tool. I think it's important for us to understand what really makes AI valuable. And this is that, don't fall into a trap of rushing to market with particular AI features.
Speaker 2:Start with a plan, have a short and long term plan. And that's going to involve things like infrastructure because AI isn't going to go away. It's very much here to stay. There are so many areas of value that it can add. It's really just, how can you align yourself with those areas?
Speaker 1:The number one thing kind of to look at first is that pattern of AI failure. It's more features over substance. So, you know, the failure isn't random. The same story kind of plays out, you know, you get a flashy demo, executives are impressed, but it's really just smoke and mirrors. And so we're seeing some of this, or we've seen some of this with Builder AI, they claim they had like an AI powered app development, but really relied on all humans and now they're in bankruptcy.
Speaker 1:And this happened this year. The same with generic AI, basically their cheaper user friendly alternatives like ChatGPT often outperform expensive enterprise contracts and usability and adoption.
Speaker 2:That's another important point to note is just, again, the speed at which this is, you know, AI is advancing. And you look at a lot of companies may come in offering sort of an enterprise contract where you have custom built AI tools for specific teams or workflows, maybe custom built integrations as well, CRM or your learning management systems. And then you've got the branding support and security compliance goes along with that. And often these solutions just turn out to be thin wrappers over existing models, like maybe GPT or Claude. They're introducing clunky interfaces.
Speaker 2:And really just the main problem is lagging behind these generic tools, which have these sort of publicly available versions and they just lack flexibility, personalization, or being able to really do things like prompt tuning.
Speaker 1:So when you're saying like enterprise contracts with our audience, I want to try to get into that just a little bit. Is it just your generic AI agents that are out there? Is it something more than that? Is it something that businesses are leveraging through just ChatGPT and then building custom interfaces over like that or Clogd? I mean, can you get into that a little bit more?
Speaker 2:Yeah, it's about aligning yourself again with the best infrastructure and the tools and not going down a bigger contract that is only going to realize value in a year plus. The problem is, if you're building, you start work now, if you're locked into certain models or certain interfaces or certain features, those are going to be not obsolete, but greatly improved by the time it's delivered in, let's say, a year's time. So it's really being aware of if you're engaging in a certain enterprise contract, bringing in contractors to do certain work, make sure that it's flexible. Any infrastructure that's being developed, you need to be able to switch out that model with the latest version that comes out. You need to be able to leverage those features that are honestly being developed by massive companies that their sole purpose is focusing on improvements to chat interface.
Speaker 2:So maybe take advantage of those generic tools and plug them in where you can and find out where you can realize deeper value and don't go down a sort of one way street where at the end you've got something that's outdated and hasn't been designed with very much a future state in mind. I think also looking at recapping on the problem of kind of features over substance is this problem in chasing features, looking at core infrastructure. And there was an MIT report that had an interesting quote, Customized enterprise AI tools fail 95% of the time, just what you were referencing earlier, due to brittle workflows and poor adaptability. Over 42% of companies actually completely abandon AI pilots by the 2024. This is disappointing because there is so much value and it's just showing the poorly executed or poorly planned projects.
Speaker 2:If we look at something which was a report by the Boston Consulting Group titled Where's the Value in AI? They found that only 26% of the companies that are actually adopting AI moved beyond that pilot or the POC stages to actually create real value, with that other sort of 74% not actually achieving any discernible value. And you can sort of put this down to a number of different failures, but you've got things like siloed work, poor data that you're using with the AI, things like misaligned governance, and also the inability to scale or to plan for that necessary scaling.
Speaker 1:Yeah, I think before we jump to the next thing, I think some of that where the whole subject or title of this podcast is the AI competence mirage. And I think the failure rates of these projects and POCs are pretty high. And to dive a little bit deeper into just the name of that, the competence mirage, I always felt that when you're, say, a designer and you're trying to develop something, code back end, front end, whatever, and you ask AI for it, your knowledge gap of that coding style or whatever it is that you're trying to do is pretty vast. I know a little bit of front end code. I know maybe just some database things here and there, but that's it.
Speaker 1:But then when AI does it, it feels so magical. And so my gap is pretty wide. But then when I have AI do it, I'm like, oh, I don't need a developer anymore. And so I feel like that's the same on the opposite end. But when you ask an expert, what is the output that AI produced?
Speaker 1:You're like, yeah, it's okay. It's not what I would ship out as a final product but this is kind of where I think a lot of especially executives where they maybe have not done development specifically coded per se, or have created security measures or created designs, but they kind of know a little bit about everything, but they're really good at business, but they think, Oh, Hey, we don't need such a big designer development team or security team. We got this through just our AI agents. That's where I think it becomes very dangerous and that's where the competence mirage comes out where it's like, I've heard these other companies have laid off all these workforces because AI has just replaced them. But I feel like that's the thing where people are not understanding and that's kind of where you're seeing these high failure rates are happening.
Speaker 2:Yeah, I think it's difficult as well coming back to the speed with which everything develops. There's almost a continually increasing knowledge gap between is being released and what someone's understanding is. There was a really good, plugging this podcast, but I'm just going to mention there was another podcast where a guy by the name of Doctor. Roman Jampolski was interviewed. And he's done a lot on sort of AI safety, very well known person in the industry.
Speaker 2:And he was just talking about how we're moving towards a singularity, which is quite fascinating, where the speed at which technological innovation occurs is so far beyond our ability to keep up with it that there's no way that you can stay competent. And this sort of fuels this AI sort of competence mirage as well, because the faster it goes as well, the less people understand of its true capability or how it should be. There's sort of two ways I can almost see that competence mirage going. It's believing certain things like, okay, it is competent enough to replace workforces, or we've tried it and it isn't good enough. Like we come back to that company that you referenced earlier of builder.ai.
Speaker 2:That started quite a few years ago and wasn't really using anything probably to the scale or the level of AI we have today. But someone could say, Well, that didn't work then, so AI isn't particularly good. And that's something that would be easy with the speed at which things are moving. Maybe you did a POC in 2024, so your bias on the negative of that, the competence like, ah, no, it's useless, it'll never do anything. But now it's completely capable or it's overcapable and it's not quite there yet.
Speaker 2:I guess it's about finding that balance. It's about grounding yourself, working with people who understand AI and can say, well, it is really good for these applications. And this is where we stand today with it.
Speaker 1:Yeah, that's a good point because it isn't perfect. And I think that's where ChatGPT did have the version five and then they people were complaining because they actually liked the prior version better in certain aspects and the way it would presented things. I mean, I see it myself when I use Claude where I've asked certain questions and it would hallucinate completely incorrect things. And I mean, you're talking about simple tasks where I ask it to scan a page and then it would make up things. And I noticed it because I did read the article or whatever and I'm like, I don't remember reading that part and it would just make it up and you would ask correct itself and it would say, Oh yeah, I did make a mistake or whatever.
Speaker 1:And with math equations that my son uses, I mean, he even says, No, it actually checks GPT gets it wrong. Claw gets more right, accurate. So there's things that if you're not checking, I mean, it's imperfect, it's not a perfect system. So you have to have those checks and balances. If you don't, yeah, we get to a point where it's like, you start to like, your gap becomes greater and greater because you're relying so heavily on it.
Speaker 1:But then if it's wrong, you are confident in the output, but then it becomes incorrect and how that moves through the system like your company can be detrimental.
Speaker 2:Yeah, absolutely.
Speaker 1:I think let's get into a little bit more of the infrastructure like done right from basically feature to function. None of this is inevitable. I think there's great examples, show how infrastructure investments actually pay off.
Speaker 2:Absolutely, I mean, are quite a few kind of real world success stories and especially with companies with domain specific infrastructure and not just those flashy demos, those are the companies that are really winning. And I think it's also about empowering teams closest to your workflows, which will kind of yield much better return on investment than just having a sort of siloed centralized innovation team. And we're seeing it with companies we're working with, where you want to identify the opportunities and you want to put the technology closest to where it's going to be felt. And that's often with those teams directly to realize those opportunities you've identified. So some of the things that we look at when you're assessing AI and looking at infrastructure as well, is finding out where can we identify those areas of opportunity?
Speaker 2:Where are the efficiencies that can be realized through building this infrastructure? Where are the opportunities to improve things like UX or CX? And how do we think, this is one of the most important ones, how do we think for the future of where this technology is going? Because again, coming back, I always come back to the pace with which this is changing, but it's important because planning is rendered null or void if you're not thinking about where this is going in the very near future. So think about how your interfaces are going to change, how interfaces can even become deprecated pretty soon in favor of protocols to just directly interface or allow agents to directly interface with what you're working on.
Speaker 2:So have that plan, start small, make it scalable, have that future state in mind, be able to pivot and adapt. Ask the questions for your infrastructure. How are we going to be centralizing any sort of data or knowledge that is going to allow AI to excel. And that's also going to be one of your competitive advantages, is your business logic or your data that is potentially your area of value. So how is your infrastructure going to adapt to allow agents to access it, to think about, you know, if you've got a web product actually a time beyond the interface that you're used to, and what is that going to look like and how can you build to support that?
Speaker 2:And lastly, just coming back to the flexibility of that architecture and just really, are you enabling the things that we talked about earlier? Models, of wasting money potentially training your own model, can you leverage out of the box models that are being invested heavily by domain specific companies in the AI realm and just being able to plug your data into those and really just being able to pivot to new features and changes as those roll out.
Speaker 1:I'm curious. In video games, cheat codes let you skip months of grinding to unlock special abilities instantly. Have you ever wished for something similar for your software challenges? What if there's a way to instantly access twenty plus years of specialized expertise instead of developing it all internally? What if you could solve in weeks what might otherwise take months or years?
Speaker 1:Would you agree that most organizations faced a steep learning curve when implementing new software solutions? At my company, Impact, we serve as that cheat code for companies looking to transform complex software into intuitive experiences that users love and that drive real business results. Would it be valuable to explore, and how might this work for your specific situation? Visit impact.i0 for a free strategy session focused on your unique challenges. Let's get to the right questions framework.
Speaker 1:Let's cut through all this hype. So let's mark the moment where talk becomes intentional strategy.
Speaker 2:Yeah, I like it. And that's where at Impact we've created this framework and we're continually enriching it to ask specific questions and questions like, do you even have AI or is it just a chatbot? How do you actually quantify that you're utilizing the technology efficiently? And what unique data do you actually own that others don't, that becomes your sustainable competitive advantage. And by doing this, does it establish that competitive advantage?
Speaker 2:And then also at a much more detailed level, deep workflows are you tackling? So it's not just superficial features, what are you actually changing? And again, coming back to the points on what opportunities have you identified that you could realize through building this infrastructure and actually addressing those pain points. So if we look at some, we talked about real estate earlier and sort of having the real estate AI versus a generic GBT feature. And that's really where the infrastructure becomes important.
Speaker 2:And we'll look at some of the things that more sort of domain specific AI can actually add onto specific systems like real estate. And then if we're coming back to that builder AI, where it's just over promised AI and under delivered results. So were they actually selling anything? No, there wasn't the infrastructure, maybe even at that stage of technology to allow them to actually deliver. So yeah, really just seeing what is real, coming back to the real estate again, like the example that you were given where it really was just a thin wrapper over GPT and being able to see that and identify, is it infrastructure developing or is it just these hyped features that aren't really adding superficial value?
Speaker 1:Yeah, and I think this is what you talked about briefly is just that domain expertise wins. It's the generic AI really doesn't cut it anymore. They're cute, but you really don't break through the devalues and the specialized domain understanding.
Speaker 2:Yeah, absolutely. And that's where you look at things like healthcare, making sure healthcare AI is, or leveraging healthcare AI that's HIPAA compliant, that's medically literate, that can actually improve your clinical workflows. Or coming back to real estate again, making sure that you're embedding MLS data, you're getting legal compliance from this AI and also assisting with things like deal flow. So if you invest in the right architecture and you do that from the beginning, it's going to give you a massive advantage or competitive advantage over other companies that are leveraging these more sort of superficial tools that can't scale or differentiate?
Speaker 1:So, yeah, I think these are all great points. And we actually, you know, I'm trying to do something a little bit different where we're actually have potentially questions that we think of that the audience may have who are listening to our podcast. And here's some like kind of Q and A that we've come up with. And I think one of the biggest ones to start off with is, is your company ready for AI?
Speaker 2:Yeah, I think that is sort of a check for scale firstly, real pain points, and also that your leadership sort of sees AI as an investment and not a toy. So what do you mean by scale and how should you look at it? Well, if we look at AI infrastructure, it's only really going to deliver a return on investment when there's enough data or sufficient workflows to actually automate. So if we think for example, well, if you're a customer, you have a customer success department or of support, you're a support company. If your company is only processing sort of 10 tickets a month, well, I don't think you really need to look at infrastructure.
Speaker 2:But a company handling 10,000, 100,000 support requests a month, well, there's absolute return on investment there. And there's sort of quick, tangible wins that you can do by actually getting the right infrastructure in place. The next point is real pain points are obviously they equate to real return on investment. So if you ask questions like, well, what is costing us things like time, costing us money or affecting customer satisfaction? So if we look at an industry like healthcare, you think clinicians spend hours on admin charting and something like that is perfect for AI automation.
Speaker 2:And if you switch across to something like retail, just processing returns is a massive cost sync. So being able to add workflows there and realize those savings is where you really determine whether you're ready or not for AI. And also how AI can solve bottlenecks. So they're not just providing insights, but actually automating flows helping you just be more performant and more efficient. And then lastly, coming to that leadership mindset and having the right one is critical.
Speaker 2:So companies that view AI as well, great, this is a cool gadget. We can sort of build flashy prototypes, you know, and, you know, well, those didn't work out. Well, we'll have to abandon them. That's something that you wanna avoid. You'd rather have a mindset, a leadership mindset, one that sees AI as an infrastructure investment.
Speaker 2:Again, coming back to the point that AI is not a trend, it's not going anywhere. This is often what is going to be a replacement for a lot human resources, unfortunately. And this is where, how can you firstly utilize it to make your existing workforce more efficient? And if you can find that, then you're ready for AI and you allocate the budget, talent, and obviously the strategic alignment to that.
Speaker 1:Yeah, so the next question is what is the difference between AI features and AI infrastructure?
Speaker 2:That's what we've been talking about previously. And I think it's good to think about that question. Like, am I looking at a feature or is this an infrastructure investment? And we can think of descriptions are really things like features. So if you look at surface level application of AI, things like generating, well, can you summarize my email?
Speaker 2:Can you give me a description of this house that I'm posting? Those things are features, they really light. What you want to look at is the infrastructure of what is a full automated workflow from maybe lead or lead generation to closing that sale. That's gonna be your infrastructure. So your features and identifying features are much more shallow and they're easy to replicate.
Speaker 2:So like that AI generated property description, it's cheap, it's available in any of your kind of AI things like ChatGPT or Claude, Canva, you're not really gonna have a competitive advantage there. It's just really wrapping, thinly wrapping a prompt from any AI. On the other hand, you look at infrastructure and it's much deeper and more integrated. So, you know, look at an AI that maybe ingests leads from multiple platforms, then could score them based on, you know, the prior conversation data, can suggest next actions, can automatically generate contracts and then maybe sync with your CRM and even your accounting software, and then finally track outcomes and continually improve. I think there's so much there that justifies the spend on that infrastructure.
Speaker 2:That's something we're doing with the clients as well that we're working with is saying, right, how do we measure the success? If we're unlocking all this data, these insights that you would have never had before about potentially customers or your workflows or your processes, how do we surface analytics? How do we understand more about that? How do we make an end to end flow that you can actually continue to improve and realize value from this infrastructure investment?
Speaker 1:Yeah, so I think a good analogy to kind of understand that is like a feature is a vending machine and infrastructure is the entire logistics chain to restock, to maintain, to analyze and evolve the machine fleet.
Speaker 2:That's a nice way of looking at it.
Speaker 1:Yeah. And I think, you know, some, some good tests for infrastructure, does it have long term data loops? Is it embedded into mission critical workflows? And would it take a competitor months or years to build?
Speaker 2:Yeah. I mean, that's it as well. What is that competitive advantage? Can you start adapting now? Can you see where this is going?
Speaker 2:You've got a clear vision for the future. Are you clearly seeing kind of that or are you suffering from that AI competence mirage? Or you know, do you have a good grounded take on it?
Speaker 1:So I guess is, how much do we need to invest to make this happen?
Speaker 2:That's always the tricky part, but,
Speaker 1:you know, I think for
Speaker 2:any infrastructure, you've really got to look at your six to seven digit figures. The infrastructure return on investment is going to you're seen on average in sort of twelve to eighteen months, whereas features you could spend on, but they vanish quickly. Why does it cost more? I think it's probably evident, but we'll go through the points anyways, you know, why infrastructure costs more, but ultimately pays off in the long run. You've got to look at aligning with the right AI integrators, engineers, product teams.
Speaker 2:If you are concerned about your sort of AI competence mirage, or you may be seeing that mirage, you know, it's best to get knowledgeable people working with AI in place to level set and understand how the AI of today and the AI of tomorrow is going to best work with what you're trying to achieve. You've got to look at starting to manage your data pipelines, also your security, training AI on proprietary workflows. Does it make sense to train directly? Does it make sense to use existing models? How are you going to benefit from those?
Speaker 2:And then integrating across platforms as well, things like Salesforce, Slack, your internal APIs. So, you know, an example is you look at something like Salesforce's, you know, Einstein GPT, and you think, alright. What are you going to get from that first, potentially, even leveraging Salesforce over their APIs, but having a custom trained copilot that is tied to everything you may have in Salesforce, but also knowledge in your company, support history, bringing all that together, that is where you're really going to realize sort of true value from these. I think looking at features can also be deceptive. You see something, wow, what is this tool doing?
Speaker 2:It's $99 a month. It's great, it does all these things and often feels like a smart choice, but you're not really solving anything deeply. And often you're getting into that trap of, well, you can't control the model, you can't control the prompt, you can't control the changes, you're tied into the platform. There's so many things that just going for those quick feature wins with off the shelf tools are problematic. We imagine something like an AI email summarizer.
Speaker 2:It's going to help individuals, but unless it's tied to CRM actions or maybe your ticket closures, it's not really adding any enterprise value there. Then lastly, you've got to look at what is your return on investment and the timeline specifically around that. So we've mentioned earlier, you look at infrastructure. Usually, it's realized in sort of a year to a year and a half. But your features are, they're cheap upfront and they're often sort of abandoned in three to six months.
Speaker 1:Yeah, so I think it's almost like hiring a contractor to either build your house or build you a house or renting an inflatable tent.
Speaker 2:Yeah, that's it. I mean, is. It's getting something that seems like it's offering you a lot, but you realize, man, first storm hits, this tent was a pretty bad idea.
Speaker 1:I think this is probably a good place to kind of close out for this podcast. And I think, basically, most of the so called AI is just AI lite. Winners invest in infrastructure, not illusions. And if you're really thinking about AI for your business, you really have to start by honestly answering these questions that we discussed. You wanna ask the right questions, focus on the domain and then scale, not just the hype.
Speaker 2:The infrastructure is what you should be following. And I think if you're chasing mirages, you've got to stop and you're going to decide, do you build or fuel real kind of infrastructure base future of AI.
Speaker 1:Yeah, and I think this goes along with, again, if you're looking to produce some type of product or feature or whatever the case is from a software perspective, we have our processes where we really do the strategy upfront first. We try to make sure if this is something that does make sense, and is this the right approach before you start to invest a lot of time and energy into something that maybe your users don't really need or want.
Speaker 2:It's very true.
Speaker 1:Yeah, I mean, definitely what you should do is, you you can reach out to us through our website if you want to have real conversations about, you know, AI and infrastructure. Leave a comment in our podcast here and ask us questions as well. Yeah, thanks for tuning in this week and looking forward to our next conversation.
Speaker 2:See you everyone.
Speaker 1:All right, take care. Bye. Have you ever played a video game and discovered a cheat code that instantly unlocks abilities that would have taken months to develop? I'm curious. What would it mean for your business if you could access a similar cheat code for your software challenges?
Speaker 1:What if you could bypass months of trial and error and immediately tap into proven expertise? You know, I've noticed that many organizations spend years developing specialized software expertise internally, often through costly mistakes and setbacks. Would you agree? That's a common challenge in your industry as well. At my company, Impact, we function as that cheat code for companies looking to transform complex software into intuitive experiences.
Speaker 1:Our clients gain immediate access to twenty plus years of specialized knowledge and the experience of launching hundreds of software digital products in many different industries without having to develop it all internally. You might be wondering how does this actually translate to business results. Well, companies we work with typically see go to market times reduced by up to 50%, their overall NPS scores rocket up, and their product to development team's efficiency significantly improved. Instead of struggling through costly mistakes, they accelerate directly to solutions that work. This is why organizations from startups to Fortune 500 partners with us for years.
Speaker 1:We consistently help them solve problems in weeks that might otherwise take months or years. If you're responsible for digital transformation or product development, wouldn't it make sense to at least explore whether this cheat code could work for your specific challenges? From boardroom ideas to code, this is what we do best. Visit our website at iiiimpact.io. You can see the link below to schedule a free strategy session.
Speaker 1:It's just a conversation about your unique situation, not a sales pitch, and you'll walk away with valuable insights regardless of whether we end up working together. Thank you.