Subscribe
Share
Share
Embed
Sheamus McGovern founded ODSC roughly twelve years ago and now splits his time between the conference business and a role as venture partner and Head of AI at Cortical Ventures. His book, The AI Skill Flip, came out of a pattern he kept hitting: data scientists and software engineers coming to him asking whether AI was going to take their jobs and what they should do about it. He wanted to write something that sat between the doom narrative and the utopian one, both of which he thinks are wrong. The "flip" in the title is the observation that the balance of skills has shifted rather than disappeared. Five years ago a software engineer spent most of their time writing raw code. Now much of that time goes to judging and evaluating what the model produced and thinking further up the stack. The same flip applies in marketing, where the skill becomes knowing what good looks like and what persona you're targeting rather than producing the asset yourself.
Asked what separates people who get real value from AI from people who don't, Sheamus lands on three things. First is passion, the plain will to get a good outcome, which he compares to what separates a strong startup founder from an average one. Second is creativity, which he argues AI increases rather than eliminates, because models are sycophantic and will happily build exactly what you asked for. His example is watching people reach for Replit, Base44, or Lovable and build a dashboard, when the real question is whether a dashboard is even the right artifact in a world of agentic workflows. Dan pushes the point further, noting that dashboards existed because software was expensive to build, so you built one thing and maintained it. Third is judgment. AI is excellent at producing output and terrible at judging its value, which Sheamus frames as another instance of the automation paradox.
On whether judgment can be taught, Sheamus starts at the engineering level with evaluations. Traditional numeric metrics still apply, but open-ended evaluation is the hard part, and he watched engineers struggle with it while building his first RAG and QA systems. His QA team's honest response was that the system was generating text all the time and they had no idea how to test it. Above the engineering layer, judgment comes down to domain expertise and knowing what good looks like. He tells a story about generating a thirty-page contract with Claude Code, sending it to his lawyer for a quick review, and getting billed ten hours anyway, because the lawyer still had to read every word and apply their own judgment. The other half of teaching judgment is teaching people how AI actually works, so they neither trust it completely nor dismiss it. Once you see it as sophisticated pattern recognition rather than magic, the failure modes become predictable. It works well on established APIs and badly on new libraries. He'd asked Perplexity that morning for the top twenty personal AI frameworks and got Gemini and other Google products instead of OpenClaw and the other recent entrants.
Dan asks whether prompt design still matters now that agentic loops can ask their own clarifying questions, and both agree the vocabulary has moved faster than the practice. People stopped talking about context engineering and started talking about the harness, but skills and memory are still context engineering wearing different clothes. Sheamus's view is that prompt engineering, context engineering, and skills are all the same underlying muscle, and the people who learned the first are quick at the third. Most teams are still doing a poor job of it, partly because the models are good enough to paper over sloppy input and hand back generic, unoptimized, expensive results. He describes users bouncing from the $20 plan to the $100 plan and still running out of tokens with no idea where they went. The deeper problem, and the one he spends a chapter on, is that knowledge work is both open-ended and unstructured. You get stuck debugging, a five-minute task becomes two hours, and workflows you assumed were deterministic turn out not to be. His own research pipeline pulling papers from Google Scholar and arXiv keeps hitting that wall, because author institutions appear below the names on one paper, on the left-hand side of another, and not at all on a third.
His own use of AI changed substantially over the two and a half years he spent writing the book, which included interviews with about thirty people. The first version of Cortical's VC sourcing pipeline was hand-architected, with AI slotted into specific stages, ChatGPT or Claude to read reports and Perplexity to search. Now a single agent with the right skills and system prompts can do the whole thing end to end. He runs a personal assistant built from Claude Code and a bit of Codex that pulls attachments from his Google Calendar, cross-references his CRM and the Crunchbase API, and prepares the monthly list of two hundred startups he has to review. What used to be three hours of review is now thirty minutes of review plus a couple of hours improving the skills, which he readily admits is not obviously a time saving yet. He's also building a speaker CRM for ODSC, which has around eight thousand speakers in its database and about a thousand submissions per conference, using planning mode and Replit, sometimes adding features live during the meeting where they're requested.
That leads to the conversation's real tension. Sheamus calls it the AI credit card, and warns that AI technical debt is piling up fast. He is emphatic that this means more engineers, not fewer, ranting about a post he'd seen that morning claiming software engineering wouldn't exist as a role by 2027. If you can produce code at 10x or 100x, someone has to evaluate, judge, and maintain all of it. Dan agrees on maintenance being the dominant cost but presses on a different point: not every piece of software is worth owning. Sheamus pushes back and they agree to disagree. His counterargument is that the gatekeepers on software development are gone, and the spreadsheet, still the most-used application in most companies, is going to be replaced by generated software and generative UI, interfaces that appear when you ask for them and get thrown away afterward. Dan draws the line at company size, where a startup should build and a large company already paying for tools has to weigh marginal value against maintenance cost.
On junior engineers, Sheamus introduces the idea of cognitive debt alongside technical debt. Every task you offload, you also stop practicing, and he offers himself as evidence that his Python is worse than it was two years ago. His advice to juniors is uncompromising: learn the classical skills anyway, including how languages handle memory and which libraries matter, because that's what makes you good at reading and judging generated code. Then learn how AI works on top of that, which he thinks makes juniors potentially more employable than mid-level engineers because they can be AI-native from the start. He's also more hopeful than most about the junior hiring market, attributing much of the slowdown to pandemic over-hiring and, per a study he'd read, possibly to remote work leaving nobody in the office to train new staff. He closes on what he wants from companies, which is real training rather than buying a tool and declaring victory. His model is a matrix: universal skills like prompt engineering, governance, risk assessment, and evaluation across the top, then domain-specific practice underneath. He also makes a pitch for data literacy, having watched software engineers stare at a loss function or an R-squared with no idea what to do with it.
Chapters
⠀
Links from the show
--------------------
⠀
Guests
-------
Sheamus McGovern, Founder, ODSC
⠀
Follow the podcast
-------------------
⠀
Follow Dan Gerlanc
-------------------
The podcast about Agentic AI and Software Engineering. Each episode is a conversation with people whose daily lives most intersect with AI and agentic systems. Join me as I follow the stories, the behind-the-scenes, and the real people behind the code.
I am Dan Gerlanc, and welcome to Agents and Engineers, the podcast about agentic AI and software engineering. Today, Sheamus McGovern is joining us. He is the founder of ODSC, a venture partner at Cortical Ventures, and most recently he wrote the AI Skill Flip. Sheamus, great to have you on the show. Thanks for joining us.
Dan, I'm delighted you started this podcast and delighted to be on the show.
Yeah, great to have you. And we've we've known each other, I guess almost twenty twenty years, I think.
Yeah, yeah, from the Boston finance tech scene.
Yep. So it's all it all comes back together. So tell me a little bit about your book or a lot about your book. what does everyone need to know? What's the thesis?
Yeah. Yeah, I'd love to do. or so. so as I said earlier, delighted to be on this show. And as you mentioned, I founded ODSC about twelve years ago. I think you were there, Dan, for one of our first events and first meetups. I'm recording this from CIC in Cambridge, on Third Street and Kendall Square and up the street is the the boiler room, I guess, which was the first meetup was held in back in 2012 So going back way back then. But yeah, so it's been quite a journey. And I wrote a book which was published just in mid-March called The AI Skill Flip. And the subtitle is How Smart Professionals Are Using AI. And yeah, like how the book came about was I was you know, ODSC AI, we we still host meetups, we still host webinars, we have our big conferences, ODSC, East and West, of course. And as part of my role at Cortical Ventures, I also interact with a lot of startups. And what struck me was two years ago, I was hearing a lot of negativity around AI. Well, we're even we're hearing even more negativity around AI at the moment, but this was more around the lines of Is AI going to take my job, or how do I change my current skill set or my current role to deal with this new wave of AI? And kind of what struck me was even more so, data scientists and software engineers were coming to me and saying, what should I do with AI? how could how should I change my career? And I was kind of taken aback a little bit about that because I'm like, First of all, if you're a software engineer or you're a data scientist, you should already be using AI in your work. And also remember two or three years ago, you were seeing people post very silly things on LinkedIn. Like, do you remember that viral post, that viral LinkedIn post where somebody said half the outsourcing jobs in India will be gone by twenty twenty five, right? Absolutely stupid commentary stuff like that. And let's face it, The stupid, the silly commentary has not stopped if anything's increased. But that kind of inspired me to write a book.
Now some of those comments are being written by AI.
Yeah, sure. AI slop reinforcing AI slop. But I wanted to to write a book from maybe a little more less hype, less doom narrative because 'cause and still today you see a lot of hype, a lot of doom, right? You look at the news any given day, AI's gonna take its jobs, more AI more layoffs announced because of AI are This utopian future where we don't have to work anymore and AI is going to do all the work for us, right? Or you're a software engineer, you can sit back and let AI code for you. We both know Dan that's not that not the reality. So I wanted to write a book, a practical. Yeah, exactly. Not yet. Anyway, I'm still working on that. So I want to write a book that really helped ground people and actually what AI is capable of, of number one, what is it capable of?
Not in my experience.
why both the doom narratives and the utopian narratives really are not going to be reality. I wanted to also tell people why the both the Doom narrative and the Utopian narrative I were both wrong, I believe. And then in addition to that, what can I do in practical terms to prepare myself for the AI we currently have? And as AI continues to improve, what skills do I need to focus on? what skills do I need to build? And that's kind of where the name came from. I was surprised how hard it was to. Pick a a name for the book, because I guess all the good names were already taken. So, but the skill flip was still pretty good because think about it. if you're a software engineer now, for example, five years ago you were definitely spending a lot your time writing raw code. And most likely now, if you have made the shift to AI, you are using AI to write the code, and then you're spending more time, A, judging the output of that code, B evaluating it. And see thinking about higher up on the stack, right? So that's a skill flip for software engineers. If you're in marketing, it doesn't matter if you're doing still images now, carousels, video, copy, you're probably using AI for all of those. Now your your skill is like, what does good look like? What are these, what's the kind of output I'm trying to capture here? What's the persona I'm targeting? What's the vibe I'm looking for? So This some skills you need to increase your knowledge in those skills, other ones have kind of decreased. So that's why I called it the AI skill flip. So I apologize. That was a very long answer to a quite quite concise question.
So what do you think separates developers or people who are getting a lot out of LLMs and AI versus people who aren't?
Yeah, I've been thinking about that a lot. And I I can give you the standard answer and then I can give you the the non standard one, which I've been thinking about because it's almost like what separates a good startup founder from an average startup founder, or what what separates a star employee from an average employee, right? And I really think it comes down to something we don't talk about enough. It's it's passion. And Passion to work, are you really driven to get the best outcome for whatever goal it is you're trying to achieve? So let's say I'm a programmer, right? And as you know, Dan, I've always been hands-on and developer. right now for Cortical where I'm also Head of AI, I've got two or three software projects I'm working on. for ODSC, where I'm the founder and I help with content and lots of other jobs, I'm also working on three or four other projects. Now I pretty much stepped away from coding for at least five or six years because I didn't have time. And now again, here I am deep in coding because I'm passionate about helping board organizations succeed and they need certain software and applications to do that. So here I am back using AI and developing new applications. So I think number one is passion, because if you really want to succeed. And you can see how AI is a roadmap to that success. I think that's kind of number one. Because still to this day, I hear about software engineers who don't want to use AI to code. And I understand that. There's a legitimate reason not to use it. But think one is not no is one is is passion, just just the will to succeed. The other one I see is for people who are using AI and leveraging it quite well is creativity. Right. People think AI is gonna take all the creativity out of us because Hey, if I just been in a prompt and I ask for something, AI is going to do all the creative work. But it's quite the opposite because again, as I said before, doesn't matter if I'm coding, doing a presentation, writing marketing copy, I have to know what good looks like. So being creative and understanding one, what does the domain look like? What is good look like? How can I use AI to achieve that? Because AI, again, is pretty sycophantic, even in writing code, it will do what you tell it to do. So a lot of times you have to Be creative in writing good prompts and understanding what you're trying to achieve and saying, like, okay, what is this, what is this going to look like? I'll give you a more concrete example. So you know the concept of software 1.0, 2.0, and now what I call 3.0. Software 1.0, Dan, was what you and I spent most of our careers building, right? Whether it R or Java or Python. software 2.0 was machine learning smarter. But what I see a lot of people doing is they're using Apps like Replit, Base 44, Lovable, to write like dashboards and almost software 1.0, right? So when I work with engineers and other people, I'm I ask them, the question is, what do you want to build? And a lot of times they will come up with a dashboard or something like that. I'm like, well, do you really want to build a dashboard? What about in the age of generative AI? Shouldn't we build building something that is more attuned to the individual user number one? should we be building something that aligns with the new architecture, which is the agentic workflows. So creativity is is the second one. And then the third one I would say is
Right, or do you even need to build a dashboard or is this a one time you build a new report, right? 'Cause before you'd build a dashboard because it was expensive to build things. So you built kind of the one thing and maintained that.
Yeah. Exactly. Yeah, and also that dashboard is that dashboard is gonna have all these widgets, which is for the average user. And most average users don't use a lot of the features, right? It was trying to be all things to to all people. yeah, and then I think there's obviously what also makes someone very good at using AI is judgment, right? Because again, it's the big skill shift. AI is excellent at producing output. But it's terrible at judging the value and the quality of that output. And I know that means may seem strange when we're using A as a judge or LLMs as a judge, but trust me, human judgment is still the core feature there. It's it's it's it's again going back to the automation paradox, right? We've seen through history again and again and again, the more we automate, the more critical human judgment becomes. So I think that's another key feature of what makes people good with with using AI. And then there's other characteristics I also talk about in my my book, things like empathy, right? we're all dialing into these call centers now that are being handled by AI agents. And the agents may sound empathetic, empathetic. they may be patient with you, but at the end of the day, they can't, they may be able to fake human empathy. But they can't really. So a lot of times when we're working with people, we're working with other people. We have to understand the nuance and the context and the domain of what they're working on. So empathy is another another another large one as well. And in my book, there's a framework I use called Epoch. the E stands for empathy, and the other letters there, but another one is is is hope as well, which sounds very weird to people, but let's say you're in sales and you're trying to hit goals. And AI can help you, but it can't give you hope, it can't give you motivation and things like that, right? So there's a lot a lot that AI can't do. So apologies, Dan, I could I could go I could go on for a long time with these questions, but I think I'll I'll pause and give you a chance to ask another one.
So do you think we can teach people the judg the judgment that's needed to work well with AI?
Yeah. I think you have to because let's maybe start low and then go higher, more abstract. But think about and you see the demand. Think about in in engineering now, AI engineering, what's a key skill? A key skill is AI evaluations, right? And why is that? Well, the work it's become very easy to produce a lot of code, a lot of test cases, a lot of output. a lot of apps, right? So the value now becomes in how can I quickly evaluate the output of that. And the evaluation itself is almost based on two sets of metrics. One are traditional kind of like data science scores or test driven development scores or things like the blue score and things like that. or you might want to be using you know your loss function if it's if it's numeric output. But the other part of evaluation that's become very important is open-ended evaluation, right? So if I'm testing a new app for a language translation system or a new app for outputting customer support, how do I know if the agent is doing a good job or not? Like building that gold standard test set that gold standards set that you can test against and make sure that the answers are what you expect them to be. And Dan, you have you have a lot of experience in structured output yourself, right? Because I know you were you were involved Yeah, you were involved in that. So that's at one level, right? And that's at the engineering level. And and I and it's interesting to me because once we moved away from numerical evaluation and into open-ended evaluation.
Could you could say that?
I saw a lot of engineers struggle with that. When I was building out my first rag systems, first QA kind of systems, the end users weren't happy with it. And I would go back to my QA engineers and developers and say, you know, guys, aren't you looking at the output of this? Well, like, you know, Sheamus, it's just generating stuff all the time. How do we evaluate this? How do we test it? So one thing I think it it has to be thought at the evaluation level for sure, and then going up to the stack to Business users, personal users, I think it goes back into the evaluation, goes back into what you said, judgment, right? So I always say to my team, first simple question I always ask is, what does good look like, right? And what good looks like usually comes down to domain expertise. If I'm in the financial space or the healthcare space or the legal space, what good looks like to me is what my training is, right? And that's what People don't understand. So I can use an app now and it can generate fantastic legal documents for me. They look good, but and this has happened to me when I hadn't motive my lawyer. I'm like, hey, I generated this 30 page contract using Claude code. Can you quickly review it for me and see if it's good to go? And the lawyers are like, yeah, yeah, yeah, we'll review this. And then they come back and they bill me 10 hours. I'm like, well. That's the same amount of hours as if you as if you would have as as if you would have written it for me from scratch using some kind of a template. So sure, Sheamus, because I have to read what the LLM wrote and then use my judgment to see if it's good or not. Right. And I think teaching judgment, I think the good news is we can use a lot of skills we have already, right? Doesn't matter what domain you're coming out of. If you're in sales and marketing. In engineers and healthcare and finance, you can use the training and the learning you got from your university and from your career and all that stuff to look at what good looks like. And then the second part though, I think what people neglect and what we we really need to give training on. I try and explain a little bit about how AI works. So that way you can understand it's not magic, it's math, it's functions, it's algorithms. It's pattern recognition. And when you start to think about AI as very sophisticated pattern recognition, you start to feel see the flaws. Like if I realize I'm writing a contract and it's writing that based on everything it's studied in the last 20 years that was written on the web, that's great. But what if if if something new came out, right? or if you understand how AI works and I'm a developer, you'll notice that it works great for For established APIs and established libraries, but it's pretty terrible working with new libraries. I was trying to use Perplexity this morning at the time of recording. I was asking it, like, give me the top 20 personal AI frameworks. And I was expecting it to tell me things like OpenClaw, NeoClaw, Kilo Claw, all the different claws out there, Hermes agent. And instead of start spewing about personal agents like Gemini and Google stuff. Like from the time of its training, right? So so understanding how AI works is very important and understanding what AI is good at, what it lacks, we still have hallucinations there is also a very important aspect. So you can absolutely train that. And then if I may say so, the other bus big missing piece is as AI gets better, it makes us much more lazy at prompting and The number one skill I tell people to learn in my book is be better at prompt design and prompt engineering. Cause I can go to ChatGPT or Anthropic right now and say, hey, I need a wellness regime for myself because I exercise three or four times a week. what it should be doing is asking me other questions, like I should be telling it what my age is roughly, any healthcare history I have. What I currently has is a have for a diet, just understands the more information we put into this, the better we prompt, the better the output is going to be. So
Do you still think that's as important with agentic loops where the model can ask for additional clarification or
I think it is because I've been focusing a lot on the a the harness is the latest thing. But have you noticed how people have stopped talking so much about context engineering and moved on to the harness? I think that's part of yeah, and I think
Dep yeah, defin definitely. Or it's moved into how do you build context into the harness like skills or memory, which are all I mean, it's all still kind of context engineering. It's just a different way of thinking about it.
Yeah. Exactly, yeah. Yeah. Yeah, and I think like we complain about things moving too fast, and that's a good example where again things are moving too fast. And because when I talk to developers, AI engineers, and even practitioners, they're still trying to get their head around prompt engineering, let alone context engineering, let alone skills. And for example, when I talk to people building age agentic workflows or trying to build them, They're still trying to discover what skills are. And guess what? The people who learn prompt engineering are quite good at building skills because prompt engineering, context engineering skills are all kind of related, number one, right? And then number two, if you look at any agentic workflows, the ones I've seen and the ones I've done are very prompt-heavy in some some case or manner. They might be system prompts, they might be rag stores. There may might be some kinds of memory, but we're doing prompting on the input, we're doing prompting on the output, we're doing prompting on the tool calling, on the tool results. And then especially when we're doing the the the age agentic turns, it's it's it's getting different. So to me, it doesn't matter if it's trying to figure out where in the harness that lives, or context engineering, or prompt engineering, for the most part. Most organizations, most teams, most individuals are doing a poor job on the context side and the prompting side, only because the agents are so good now, well, let's say the LLMs are so good now that it seems like they can build all this stuff out for you. But the the the problem there, it's you're not using your skills and it's just giving you an average output, right? Because if if you were to push back against AI for coding, what's the biggest pushback? It's just giving you fairly generic code, right? It's not the bet most optimized. It doesn't have the most context. and yeah, you can use the code. I've I generally use the code it spits out, and maybe I tweak to I tweak it a little bit. But I think it's the same with using agents. We tend to use whatever the AI is giving us, and that tends to be very high in token cost. and other costs like most people to me are complaining bitterly like they were using twenty dollar Claude Cowork. And then they were putting more stuff and they ran out of tokens. And then they went up to the $100 part plan. And guess what? They're still running out of tokens. they've no idea how their tokens are being used. Now, what I tell them is look at the plot code when it's running. And look what it's doing all the time. It's just it's thinking all the time and it's doing so heavy, and no one's doing token, token, not token maxing, the opposite. Like there's lots of new techniques out there. Yeah, to token token batching.
Token minning.
caching of tokens, caching of answers. So yeah, I th I think we're very early in people learning how to use AI effectively, very early. And in some cases the harnesses are running way ahead way ahead of what people are able to use them for.
Yeah, I think using something like Claude Code, you can use it works well because it can run a loop and use some huge amount of tokens, but most workflows that people are building are if you can avoid having to run it in a loop, you're better off. So I think it's changed a lot too from before
And self improved to a certain degree, yeah.
using the API and working with LLMs, prompt engineering was kind of the same thing. And it's diverged a bit now between are you interacting with an agent harness like Claude Code or are you actually building workflows that need to operate with very clear and kind of the minimum context you can give them.
Yeah, and I and that's the other challenge I discussed in my book. if you think about your job, your role, or anyone else's role or job, what I think the AI narrative and the AI job replacement narrative gets wrong, continues to get wrong. Any given job, any given role in knowledge work, at least anyway, tends to be have two key characteristics. number one is They are open-ended. Your job is open-ended. Meaning you can spend sometimes five minutes at a task or a couple of hours, right? And I can talk a little bit more about that. But they are for the most part open-ended. And the second part, your job or role is unstructured, which sounds very strange to people. It's unstructured. Well, when you think of it this way, you wake up. Turn on your laptop, start your day's work. And for a lot of us, our days are very structured in the sense that I'm looking at my calendar, I got this meeting, that meeting. I go to a stand-up, I tell them what I did yesterday, I'm gonna tell them what I'm gonna do today. But your your day ends up being a little more unstructured than that, than that, right? You tend to get stuck in rabbit hole, stuck debugging, stop working on something, something you thought that was gonna take five minutes ends up being an hour or two, right? So. So because a lot of our work is both unstructured and open ended, when we try and build agentic workflows on top of parts of that, it doesn't quite flow the same way we believe it to, right? So for example, some of the workflows I've built for Cortical are deep on the research side. I want to be looking at Google Scholar. I want to be looking at archive. I want to see what the topics people are writing about. I want to know who those people are, who wrote those papers, what institutions they are. And that for me is a great example. I can use Claude Code or something else. And if you ever read these papers, sometimes the institutions, the authors are, it's listed below their names, sometimes on the left-hand side, sometimes is missing. And Building a workflow which may seem like it's something simple ends up being something that goes from what you think should be deterministic into this the non-deterministic loop. And the added challenge is it may work one day and then all of a sudden it doesn't work the next day. And I feel a lot of people are seeing that with Claude Code. And guess what? Even some of these workflows become open-ended and unstructured in the sense that they don't always deliver the structured output. That you hope for. So so my book kind of dives into that a lot. That if you understand the nature of work and what you're trying to automate, it helps understand that because AI, for the most part, is still very good at very structured, repeatable tasks. And you'd be surprised about how little of our work is non-repeatable and not structured.
So how are you using AI differently than when you first started writing the book?
yeah, so I started writing the book Two and a half years ago. And
Because I knew how AI was going to be become more care more capable, I didn't want to write a book that was just my thoughts. So the main thing I did was I interviewed a lot of people. So I have about 30 people in that book that I interviewed. And it was interesting because I was using AI for doing some code work, but not a lot. But I wasn't using it for anything personal at the time. And when I interviewed this people as well, for the most part, I interviewed people like Matt Harrison, who was using it to help him teach some of his class. I'm not sure if you know Matt, he's an instructor at ODSC. So yeah. And then also at the same time, I was building out Cortical's first version of our venture capital pipeline for finding new founders and startups. And what I was able to do very well.
Yep. The Py does a lot of Python. Yep.
was as you mentioned before, if I gave it an API and wrote some code against that and pulled in things like reports and data from like pitchbook and all that stuff, I was able to use something like ChatGPT or Anthropic to understand what the papers I got were. And then I would use something like Perplexity to go ahead and say, Now do a search. So I built a stack based on AI and in in using AI in different sections. Now what's different is you can obviously run claud code to do to to do all of that, right? I no longer have to build that Frankenstein code with AI kind of in certain key parts of it. Now you can basically ask AI to do that. So for example, you could use claud code conceivably right now to go Tell me, go go search the web, which which startups have recently launched, or let's say which startups have joined Y Combinator. You know, if they joined, that means they don't have any seed or pre-seed. Find out the ideas, find out the founder, who the founders are, go look them up, see what they did before, have they ever had any successful startups that they previously launched? which of these startups meet my investment thesis? And go look for their LinkedIn profiles and contact information. Now you can basically build that based on one kind of a set of skills and a set of system prompts, right? Whereas before, I was kind of architecting AI in the stack. That's that's one way I'm using it differently. A second way I'm using it lot more for
it more through Claude Code directly as skills?
Claude Code directly, but I'm using Codex as well and I'm using other ones as well. But I'm also using it more of an A as an AI assistant. Cause before a couple of years ago, it wasn't good enough to go through my emails and surface the right ones. I'll give you another good example. I'm I I go to this event every month and they send us about 200 startups in my calendar link. Before what I used to have to do was go to my calendar, download the list of Startups they wanted me to review, go into Crunchbase or PitchBook, look at them. And maybe a year ago I was able to use Perplexity and Computer Use to say, Hey, here's the spreadsheet, go use it. But now what I'm able to do is basically my personal assistant can go to my Google Calendar, pull down the attachments,
Real you're a real human or AI personal assistant.
My AI code, which is a mix of Claude Code and a little bit of Codex but not Claude Cowork And it also connects to to my CRM, which has these lists of startups, so I it it knows if I've already talked to them or not talked to them. And it also connects to the crunch base API so I can see if they've raised and what they've raised and who the founders are and all that stuff. So it's basically acting as an assistant to me, pulling in all this disparate work, which would have taken me honestly two or three hours. And now my job has shifted, my AI use has shifted from doing that work to reviewing the outputs of it, like seeing if it missed any good startups that I should not have, reviewing the output. And what I spend a lot of my time now is is tweaking the skills to say, okay, well, you got these, but you missed this, this one here. are the next time so so to me the AI has become I wouldn't say it's saving me time which is weird to say because instead of spending three hours reviewing that list ever every every couple of weeks now I'm spending 30 minutes reviewing the list and then a couple hours improving the AI. So maybe in six months or a year's time I'll get to a point where the AI is fairly solid but What's also happening as well is the AI keeps improving in capabilities. So then I ask more from it, right? because for example, it wasn't very it's like with MCP servers coming out now, I can actually ask it to go to more MCP servers and and use certain tools for doing certain work, right? So so it's interesting, yeah. So I would say the big use cases are definitely using AI to help my workload. And and then lastly I would say the the vibe coding stuff i i is is I'm not trying to hype it up like in saying it's it's as good as it'll ever be, but it is incredibly, incredibly impressive. Like in here's an example. the speaker team at ODSC has for years tried to find a speaker CRM because Dan, we've been doing this 12 years. I think we've got about 8,000 speakers in our speaker database. And then I think we've researched about maybe 30 or 40,000. And for every conference, we've got about a thousand submissions, but we still don't have a CRM. We tried HubSpot, we tried other ones, but now I'm actually basically building a CRM. And what's different is we'll have a meeting and they will tell me a feature they need. And while they're telling me the feature, I'm actually asking Replit to build out that feature for me. Right?
So have you been doing all of the engineering for it?
No, so what I'm doing is I'm using planning mode to do out the engineering for it. So planning mode is basically you tell whatever these agentic vibe coding platforms are, you tell it what you need to build. you might be using another, I I sometimes use another LLM for that. Say, here's what, here's what I'm trying to build. I'm trying to build a CRM for my specific business. Here's the indiv here's the key characteristics of it. Here's what I need, here's what I don't need, right?
Right, but you're not you're there's not a separate team that's building this out. You're
just we're just building it together. So what I mean is like Yeah. Yeah, my my design team, my engineering team is Claude Code or whatever. And you know you know in the old days when you're trying to build something, you would sit down with a system analyst, would go talk to the end user and say, How are you working with this? How are you using that? Now you're able to c compress all that down. You can sit there with the users and you can almost build the features. Well, not almost.
You're the only engineer plus Claude or
You can build the features in real time and show it to the end user and say, is this what you want? Is this what you mean? you can roll back, you can try it a different way. And I'm really enjoying that because it's it's highly motivational when you can build speed you can build code at this at the at the speed of of of thought you know what I mean? Of sp at the speed of just describing it.
Yeah, it puts it a lot closer to solving problems versus just writing code.
Now that is gonna yeah. So as we know there's no free lunch, and that's what we call the AI credit card. You keep swiping it every time I build put in a new feature, every time I build a new app. and I'm still trying to figure this out, but we are massively increasing the amount of AI technical debt. Right. Like every and and another reason why I wrote the book. Because people like I said at the very start, people were saying things like software engineering is gonna be dead as a rule. I just saw I just saw a post this morning. Can I rant a little bit? Some idiots went out to the web and said, software engineering, the role of software engineering will no longer exist in 2027. Like the most nonsense thing I've heard in a long time. And I hear a lot of people saying that, because at the end of the day, sure, the role of software engineering will change. And yes, you can call yourself an AI engineer now because you're using AI. But an AI engineer to me is someone who understands deeply how AI works, right? A software engineer, on the other hand, is someone who uses AI to build software, but all the other skills they have of building software still apply, right? And so when people say the role's going to change or we're not gonna need any of the software engineers, I've been saying this for the last five years. That is the dumbest thing that I ever heard. I think we're gonna need more. Software engineers, more AI engineers, whatever we're calling them, we're going to need more. Because if now I can produce at a 10x or 100x velocity, I'm pretty soon we're going to get to a hundred, a thousand X velocity for producing code. All that code's being produced, who's going to evaluate it, who's going to judge the quality of it, who's going to maintain it, who's going to ask AI to update it, that's going to lead to increased demand for engineering, not less.
Yeah, I agree that even pre-AI assisted coding, most of the cost of software has been maintenance of the software, not writing it in the first place.
Exactly. And no one wants to do that. It's the least loved part of software. And AI can help it there, but yeah, when you're when you're producing code at ten X, a hundred X, a thousand X, it's it's gonna create more opportunities, more role and require more training for sure.
And there's also not always a strong case to create all of the code, I think, either. Just because you can do it doesn't mean it's something you want to own or maintain longer term.
Yeah. We could agree to disagree on that. Because look look where I'm coming from. I'm coming from the startup ODSCAI vibe coding. And and you're right. Like if if you were the head of head of a company tech at a company, you would absolutely want to say that because you don't want let's say you got five thousand people in the company and all five thousand people are vibe coding this stuff and asking for access to customer data and all that stuff. You're right. But for the most part
Well, I think it applies more at it's very different, I think, for a smaller startup versus a larger company. Where I think in a startup or early stage, yeah, you do want to build this stuff. But in a larger company, you already have people who are paying for things. What's the marginal is the marginal c value of building more software higher than the cost of maintaining it? That that's the part where I'm I still am not
Yeah. Yeah. Yeah. Yeah, but I think it's kinda like Yeah, that's true. But also AI is becoming more coding and AI is becoming more personal. Software development, the the gatekeepers are off. So think about the one the the number one application up until the number one application, maybe still to this day in corporations is the spreadsheet, whether it's Excel spreadsheet or Google spreadsheet, right? I think that's indisputable. Like Everyone I know still works off spreadsheets. I work off them. Every company I go to, they're always working off spreadsheets. But what you'll see in next year or in four years and five years, that's going to change. Because now I can, instead of building a spreadsheet, I can build software around that data I collect. Right. So what they're doing is they're taking data and context from one place, putting it into a spreadsheet. But you're going to see Anybody sales, marketing, HR, they're going to start using AI for building information portals like we've used spreadsheets. And it's already moved in that direction because if you think about the advent from GAN's generative adversarial networks, to all of a sudden this explosion of transformers and how we're able to build this AI, it led to Image generation, video generation, audio generation, music generation. But now we're entering the era of generative user interfaces. So basically, like MCP came up with this as well. I can basically go to an MCP server if it has this capability and they switch it on and say, Hey, you're an MCP server for Amazon products, build me a quick dashboard because I'm looking for you know, three person tents or whatever. And that It may generate that UA UI and you may never use that UI again. Right. So generative UI is becoming a thing where show me what I want to see based on what I'm asking you in the context. And if it's a report or something like that or something reusable, I might save that or otherwise it's it's disposable. And all things in my everything I see point to the spreadsheet being replaced by code. And you're right. Like we have to solve that problem of increased technical debt. Because maybe the answer is not building all these apps, but maybe building more generative interfaces that appear and disappear based on need. And and again, I still think we're thinking about software with a software one point mindset and not a software three point mindset. What I mean by that is you're right. Just because I can build my own CRM, do I need to? Well, yeah, I need to. But you're right.
Right, because here there's nothing that exists to do it, right? You have a f unique problem and companies have built internal software since time immemorial, but before sometimes it would just be a spreadsheet 'cause building the software was too expensive or too hard.
Yeah. Yeah. Exactly. Yeah. But that's a good example because I'm building a software 1.0 application, which is a CRM. But as AI gets better and my the team members can communicate, I should have agents doing all this stuff. And there is no need for a CRM, right? And I have met people, I was just on the West Coast last week at the time of recording. I was attending an event there and afterwards ODSC AI, we hosted with a VC firm called Superset, we hosted an evening there, and I was talking to one of their salespeople. And they're no longer using LinkedIn our Salesforce, they're doing everything with agents. And yeah, so so we'll see how that evaluates. but technical debt is is a huge problem.
And I th I think building applications systems also gets back to what you were saying earlier that having the judgment or knowing what the problem is is something that AI can't tell you. You need to know what the problem is, and the advantage you can bring with it is actually working with the user directly and The A the AI can't necessarily do that.
Yeah, exactly. With the CRM we're trying to build, it's like built trying to build a CM CRM 30 years ago. we have about 120 key fields, 20 of which are required. and of those 20 that are required, there's about 30 percent missing data. And then there's another twenty percent that is just wrong data. And then there's stuff we have to backfill historically to get everything uniform. there's inconsistent formatting. Sometimes you've got the dates in certain way and other ways and like that. So so yeah, like AI can build a CRM for us, but a lot of the issues and the main knowledge you would need for building a CRM is still there, right? So so yeah, you're absolutely right. that that judgment is is is kind of key. And that doesn't go away. If anything, it increases because if we don't give if we don't know again, going back to prompt engineering and context engineering, if we don't know how to tell it to build this system. Correctly, it'll just happily go ahead and burn tokens and spew out codes. And then what you find yourself doing is what I see as reverse feature engineering. Because what AI is not, I notice what AI is not good is they sometimes roll features back partially. So if you modify your features on the fly, you kind of end up with this bit of a Frankenstein monster, which is still the fingerprints of the old features that you delete it. And it can also get confused. 'Cause AI is always sounding so confident. It's always like done, I updated this, I I got that done. It's yeah, it just requires a new skill set.
And you talk about in your book how this affects junior developers and people who are learning, where they could just go to an LLM now to get an answer versus or to build something without necessarily learning at first. How does how do we adapt to that world or how does that affect people who are not classically trained software engineers?
Yeah, big time. Huge. And you know, I'm sympathetic with with universities when it comes to this, because a lot of people criticize universities, they're out of touch, they're teaching the wrong skills. But I believe both sides of the arguments are right because number one, it is a huge challenge for junior engineers because now they kind of have to learn and train differently. yeah, because now they have to learn and train differently, because going back to debt. like technical debt. The other thing I wrote about in my book and I interviewed a lot of people about is called cognitive offloading or cognitive debt, right? So anytime you ask AI to do something for you, you're you're outsourcing that task to AI. But you're also doing a cognitive offload, right? So if AI is doing all my Python coding or React coding or it's learning a new language, Language I learned, I'm not using it. So as you know, Dan, we we tend to like my Python coding is definitely worse now than it was two years ago. Because I'm not actually doing any Python coding, I'm having AI doing it, right? So that's cognitive overlauding. you're also increasing your technical debt, your your cognitive debt, because if you're having AI do it for you, then you're not learning how to do it. So the challenge to junior engineers is they have to learn. How to use it, like they have to learn how to program, they have to learn the the the patterns, they have to learn all the different skills that an engineer two or three years ago would have had had to learn. And then also in addition to that, they have to learn how to how to work with AI and how AI works, right? So if if my yeah, so junior engineers have asked me for advice. So what I tell them is you need to learn the classical skill set that you always need to know. Like You should know how these languages work, how they how they handle memory, how they how the what are the most popular libraries are used out there. Maybe you don't have to learn it to the same extent, but you should be able to code yourself because you still have to go in there and code sometimes, right? And even if you don't do that, it'll make you much better at reading code. It'll make you much better at evaluating and judging the output of code. And then also they need to learn, and again, this doesn't have to be a two-year degree, like you can take courses at ODSC AI, you can read my book. But have at least an understanding of AI work how AI works, right? So you and I say it all the time: like if I'm trying to build a new feature into my application, and let's say I'm using a new library, like I'm trying to build in some open claw or neo claw or something like that. The LLMs, because they have a cutoff in the training date from let's say September of 2025, they're gonna know nothing about that. And yes, they can go search the web, but they're looking at different APIs. So They need to have the classical skill set and they need to understand how AI works. And that will make them better at evaluating the output of AI and also judging its output as well. So in some respects, they should be more employable than mid to senior engineers because they can become more AI native. And the other thing I would say is well, it's never been a better time to learn because AI is a fantastic tutor. Right? For software engineering, for all these skills, like basically think of any program language you want to learn, any system settings. AI can do that. Like one of the little tricks I do, instead of using Claude Code to do some of my personal AI stuff, do you remember do you know Google Scripts, Dan, Google Apps Script? A lot of people don't notice, like, but Google's got some fantastic app scripting capabilities. So it doesn't matter if your calendar, your Gmail, or your Chrome browser, or docs or spreadsheets.
Yep,
What you can do is you can create these App Scripts that run in your Google environment and around your Google stuff. And Microsoft has similar ones as well. So what I do is I use AI to code these scripts to do things for me, such as go check this set of emails, go check this link on a calendar, go check this person out. Like a lot of times I get a invo calendar invites from people. I see no email chain, I have no idea who they are. So I actually have an AI assistant go out there and look up who these people are. So what they do is they check my email, and if they don't see any conversation or no discourse, they'll try and using their email address, look up their company name and tell me what they do. Right. And actually, I don't use Claude Code to do that. What I do is I have a Gmail Apps Script and calendar script running. And what it does is it does an API call out to Gemini or to OpenAI to do that work, right? So So junior and junior engineers need to be thinking like that. Like AI shouldn't do everything for you. understanding how these libraries and frameworks and classical software engineering works. And then, sorry, the last thing I will say, so Dan, when you and I worked together, I was a software engineer, but then I was a software architect, system design engineer. Like there's a whole stack of skills that a junior developer needs to learn to move up the stack. So now, because they don't have to spend so much time. you know, years and years t home and honing their coding skills. I think they can actually move up the stack quicker. What do you think?
Yeah, I think I guess it depends what the constraints are around where they're working. Maybe maybe in if you need the code to be super high performance, then you're gonna have to spend time learning that versus if you can do less coding and learn more about how do I deploy this or build the whole system around it, then I think there's that possibility, whereas before the amount of time it would take you to learn each of those things, you just would ha you would there would no alternative to putting it in, or you would just be stuck at zero, I I feel like. So now you can get more done with
Exactly.
less knowledge and leaning on the the models to fill in where you don't know or aren't don't don't have the the depth but still can direct.
Yeah, because you know, outside of the coding, there's lots of different things that need to be done in this engineering stack that AI is not good at because I still think that, for example, everything that AI writes to me is the latency is always terrible, number one for the most part. number two, the the the token usage is never efficient. sometimes the data polls are not efficient either. Like it'll like it A good example is if if you if you ask it to query a database, it will query the database, but it won't ask you to see if you hey, you should think about doing an index on on this for this query. You know what I mean? Things like that.
Yeah, or it might write run SQL queries in a loop as as opposed to pulling ten records. I've seen I've seen things like that. Even in recently.
Yeah, exactly. Yeah. Yeah. Yeah. Yeah, exactly. So junior engineers need to learn all that. And that and you know, yeah, it I I'm hoping I'm hopeful that the the job issue will turn around where companies are starting to hire more junior engineers. And I think I think some of that is probably due to AI, but I think most of it is due to things like over hiring the pandemic. And there was just I'm actually writing an article which will be published the next few days before before this podcast came out, but com there was an article in the New York Times and actually no, I forget where it was published, but there was a study published that perhaps the remote working from home has has has affected the hiring of junior developers. Because the more that we work from home, we don't have time to be in the office to bring in new, new, new staff and train them up, same with interns. And maybe working from home has something to do with the lack of hiring for junior engineers and junior developers. But my advice to people out there is who are who are engineers is like, yeah, it's still a great career. Trust me, there's gonna be lots of jobs in this. Don't believe the doomsayers. And Dan, you and I, you know, I'm older than you, Dan, but you know, I've seen the wave of hiring you know, be either strong or soft, or strong or soft, and this is just just part of the cycle. it'll it'll turn strong again for sure.
What do you think we have to look forward to in the next six months, a year from AI capabilities?
Well, number one, I want to be able to use Claude code and not give it the same permission that I just gave it a half an hour ago. So speaking of loops, one of the things I have with computer use AI, so let's talk about it real quickly. So I'm looking forward to the computer use AI improving. So when I say computer use AI, it could be per perplexity computer use, it could be OpenAI Codex, it could be Claude Code or Claude Cowork rather, which a lot of people are using. it could be genetic workflows, because right now. from my use and other people's use, they're very clunky. They're asking for the same permissions multiple times. so number one, I think my productivity can get a real boost if that's that's more performance, right? That's I'm looking forward to that. I'm also looking forward to the vibe coding applications. become much more capable. So what I mean by that is when I first started using these vibe coding applications like Replit, Space44, Lovable, all the other ones I have to try because part of my job. they're very, very good at writing front end. They didn't work very well with databases. So for example, Lovable works great right now with I think it is Superbase, right? So it's been an improvement. But I have a bunch of applications that I've written years ago that I still love and want to use. But guess what? when I go and use Codex or Claude Code and I say, take this back end, take this database, and build me a whole refreshed front end, it's still doing a pretty terrible job. So I'm looking forward for AI being much better at using what we call brownfield coding platforms and being able to work with that 'cause that's mostly what we work with. And then the other thing I'm looking forward to is you said it Dan is We're we're building up the technical AI debt very, very quickly. And I'm really hoping that more companies will start to focus on maintaining this code because really when I build an AI application, I need an agent or something like that to to to improve that. And in addition to that, I think We are going to see much more focus on these AI skills as as as real career development right now. Cause I think too many companies, they're just throwing AI onto their workforce. So one company's idea of using AI could be, hey, sales team, you we're gonna buy Clay AI because Clay AI is AI driven. That does not make you an AI sales department, right? or hey, we're gonna introduce claud code to the whole company. Like companies really need to take skills learning much more serious, right? Because think of it this way: the if you were teaching AI or you know, if you were teaching technical skills to a company and you didn't get that right, the risk of getting it right. Weren't that consequential, right? But now companies are introducing much more powerful technology into their workforce. and if they don't do with adequate tw training, then I think they're in trouble. Because it's Dan, it's kind of like when you and I were working in in quant finance, building those models and releasing them into the trading desk. If they went wrong, bad things happened. And I think it's the same. Like, Think of all these people using Claude code, giving it access to their CRMs, customer data, giving access to their company files and all that stuff. I think I'm hoping that people will invest more in their AI skills and and and think it's not just a passing thing, then companies will follow suit. That's what I'm really looking forward to. And then maybe some of the backlash against AI will will will die down. And Maybe people will stop and and CEOs will stop thinking looking to AI as replacing humans and more as the technology to work alongside humans. You know what I mean?
What what kind of training or skill sets do you think are best or what's the best way to teach people that?
Yeah, so I think, you know, start with the basics, like build a matrix. So number one, the matrix is what skills are universal for everybody. Like prompt engineering is one. good governance is another, right? How do I use AI without harming myself or my company? how to use AI, understand the risk involved. Like those are universal. So risk assessment, governance, how to judge AI, how to ask questions, how to evaluate the output, that's universal. So that's like maybe the top row, the top columns of the matrix. And then the next part is look at it from your domain perspective, right? You may be in a regulated industry, you may be in a not re regulated industry, you may be in finance, healthcare, biotech pharma. What's best practices? What does good look like? how are other people using it? So we had a bunch of round tables at ODSC AI East here in Boston. And It was interesting to hear the different peer groups from different different industries, like the legal team was using it in different ways than say people in biotech and pharma versus people who are HR, right? so I think you need it's almost like cross-functional teams. You need individual teaching across universal skills and then domain specific ones to kind of to kind of leverage that. Because if we're going to, if we're going to teach AI to help us with our workflows, then they need to understand the nuance and the context we're working in, right? And I see a lot of people like rushing into, hey, I want to take an AI trading course. But well, do you know anything about LLMs? Do you know anything about governance and risk? And another thing, an interesting thing as well, is like another course I teach is data literacy, right? Like if you if you take data literacy, you understand things like data bias. You understand things like you know, the law of large numbers, statistical significance. You understand if you got good data or bad data, because too often we just we we we throw people into these roles and don't give them the fundamental skills. And yeah, I've been saying it for two decades, but data literacy is also very important. I don't know if you'd agree with that or not, but I see that in so many different different roles. Like even with software engineers, for example, software engineers when they're using AI and all of a sudden they have to look at loss functions and the the model saying, Well, here's the loss function or here's the R squared. They're looking at that and I'm like, Well, I'm a software engineer, I've never used that. But a statistician or a data scientist or a data person would know that. You know what I mean? Like fundamental stuff or is this data outside the standard deviation of what I'm needing?
Well, Sheamus, thanks again for joining us today. It's great to have you.
Thank you so much, Dan. Great questions. Really love your questions. And yeah, love that people could check out my book. It's the AI skill flip. It's available on Amazon and I can give you a link for the show notes.
Yeah, we'll we'll have all those links on the show notes, so you can find Sheamus and find the book.
Thank you so much, Dan.