In this episode, Andrew talks about his recent deep dive on prompt engineering and the way the AI product landscape is shaking out. Sean talks about his recent trip to Philly for a cybersecurity marketing conference. Then the guys reflect on 2024 and their goals for 2025.
Links:
Sean
two, one, happy birthday Austin, yay.
Andrew
Happy birthday, Austin!
Sean
How was your recording last week? How'd it go?
Andrew
I thought it went really well. um guess you tell me, have you had a chance to listen to it yet?
Sean
Yeah, my train from Pennsylvania, from Philadelphia to Virginia last week got very delayed, so.
Andrew
Oh no, I'm sorry.
Sean
said in the terminal and listened to it. was good. It was, yeah, thought he was super insightful.
Andrew
Yeah, Alex was awesome.
Sean
I feel like he's beginning, you were a little, what is it like? like like starstruck felt like i was like wow didn't see me like this so before it was good i thought he gave you like really good feedback i thought but i mean there were things i learned from it being someone who's never looked at the stack in the brick stuff and yeah i think it was it was helpful and then i saw the new meta monster update today or that you sent down i was like wow it is way better than before
Andrew
Oh, that's funny. Yeah.
Sean
and
Andrew
Well, that's good. At least I'm taking what I'm being taught and internalizing it and not it's not just going in one ear and out the other.
Sean
Yeah.
Sean
Right, right.
Sean
Like our podcasts usually do.
Andrew
Yeah, I tried to also do some of the pulling the slingshot back, as Alex says, some of the anticipation building with the actual release of the podcast episode. So tried to like hype that up. Austin said it worked on him. I'm not sure it made a big difference on our analytics, I think. Yeah.
Andrew
You know, I think we had. You know, it's performing better than our average podcast episode, but I think that's as much about like Alex retweeting it as it is anything else. So, you know.
Sean
Well, the real magic is, let's see how many people isn't today is right. So hello, new listeners. Austin is Andrew's co-founder, or I'm going to last name.
Andrew
Hey, that'd be cool.
Andrew
Yep. Austin's my co-founder on MetaMonster. He's the one responsible for making the product look great and function. Well, I'm just the guy who talks about it and is trying to talk about it better, trying to talk about it more good.
Sean
Yeah, hell yeah.
Andrew
Yeah. How was Philly?
Sean
It was good. I went to, so I went to cybersecurity marketing, the cyber marketing conference or cybersecurity marketing conference held by the cybersecurity marketing society. was good. A lot of, mean, no, this is the third time actually.
Andrew
Is this their first time doing an in-person conference? Oh, wow. I thought they were all virtual before.
Sean
The society, sorry, this is their, this is their third conference. They've done a bunch of like in-person events. Yeah. The society's grown. There's like 500 people at this conference.
Andrew
Cool.
Sean
yeah, I get to hang hang out with people that I don't usually get to see. And it's weird. It's, it's like the only conference I've ever been to. That's, I think that's i've ever, ever really been to that's not a security conference.
Andrew
Hmm.
Sean
So it's a whole other sort of vibe.
Andrew
Wait, but isn't it, I mean, it's still a security adjacent conference, right?
Sean
For sure.
Andrew
But it's not hackers talking about like the new vulnerabilities they discovered and like how they reverse engineered some crazy low level protocol or something.
Sean
For sure. For sure. But it's not like, you know, it's not like RSA. It's not like i can not be solid. It's yeah. Yeah.
Sean
Right.
Sean
right right exactly exactly and and it's not like it's also it's also not cyber security vendors trying to sell you on their product
Andrew
Right.
Sean
like RSA and Black Hat.
Andrew
Okay.
Sean
And it's also, it's a society it's like a society conference, so it's not, I guess it's almost like the DEFCON, like in that case, like the DEFCON of cybersecurity marketing, like you don't really go to sell anything there, you're not, your customers, I mean, I guess my client customers are there, but I also try not to do any of that.
Andrew
What does that mean?
Andrew
Hmm. Yeah. Yeah.
Sean
yeah There's obviously vendors and sponsors and stuff. We're a sponsor as well. i I literally was just like, I don't want to, like I also would like to attend the conference. So I gave up my booth and switched it out with a 360 camera and just let people use a 360 camera there.
Sean
And then went ahead and hung out with people there, but it was good.
Andrew
Oh, that's fun.
Sean
It was, yeah, I mean, would go again.
Andrew
Cool, sweet.
Sean
I went to Virginia for a friend's birthday, going to Seattle.
Andrew
Nice.
Sean
yeah yeah But you've been you It sounds like you've had a cool little rabbit hole of a week, though, if you want to share.
Andrew
Yeah, it sounds like a good week.
Andrew
Yeah, dude.
Andrew
So I've started working on my next blog post for MetaMonster, which is basically like, how do you do this today? And one of the ways you can generate metadata using AI today is using the custom JavaScript feature of Screaming Frog, which, by the way, is kind of clunky and like requires you to be someone who isn't intimidated by code because you do have to like I mean, it's not a whole lot of editing.
Andrew
It's kind of just copying and pasting in a couple of spots. But you do have to write some code.
Sean
Mm hmm.
Andrew
And it's like buried in three levels of menus and stuff. And you have to configure certain things to make it work. It's confusing. Anyway, so i'm I'm writing about how to writing about how to generate page titles and meta descriptions using custom JavaScript inside of Screaming Frog. And then at the same time, we've finally gotten our tool to the point where we can start adding new sites. And so we're testing it on larger sites that have more content. And it has shown me very quickly something that you know I've heard from other people, but it's the first time I'm getting to experience it myself, which is that like
Andrew
the AI tends to be very repetitive in the like style of descriptions that it creates. I ran Screaming Frog against the andriascans.com blog and sorted things by alphabetical order in the descriptions that it created and it had like 20 or 30 descriptions that started with discover.
Andrew
there's just There's words that it loves to use.
Sean
yep
Andrew
Discover, explore, join.
Sean
Yeah.
Andrew
you've You've pointed some of these out to me in the past. What's seamless?
Sean
Yeah, yeah. Yeah, I got a whole.
Andrew
is that Is that the one you always talk about?
Sean
Delve Delve.
Andrew
Delve.
Sean
I love Delve.
Andrew
Delve is one.
Sean
Yeah, I got a whole whole. I got a list for you. I'll just read read off really quick.
Andrew
Hit me with it.
Sean
Never mind a lot. I have no idea where my list is. All right. I'll come back. I'll put in the show notes. I'll put it.
Andrew
OK, cool.
Sean
Yeah, yeah.
Andrew
so So once I realized this, I was like, OK, it's time to start learning about prompt engineering for real, because I need to fix this.
Andrew
like If we ship the product and it does this, it's going to suck.
Sean
Yep.
Andrew
So yeah, I started diving into prompt engineering
Sean
Yeah.
Andrew
basically spent all day yesterday working through the first three chapters in the book. It's an O'Reilly book. It's called Prompt Engineering for Generative AI, I think. um And it's a really, really good primer on just like how LLMs work and how to think about prompts and how to start testing prompts at scale. And this, combined with some of the work I've been doing at miscreants,
Andrew
You know, we have a lot of clients in the cybersecurity space who are starting to build AI into their products, whether it's building agentic systems or building like tools for testing agents or building just AI systems in a little bit more in more traditional ways.
Sean
Yeah.
Andrew
It's all got me like thinking about the AI landscape and trying to, and starting to do this prompt engineering research really helped a lot of things click into place. And so I feel like I'm finally starting to make sense of how some of the pieces fit together. And a lot of terminology that's been tossed around by clients or by people online is starting to make more sense to me. so One of the things that I think is like pretty basic, but but wasn't really clear to me fully until yesterday. So all of the models that we talk about all the time, chat GPT,
Andrew
And chat GPT is really an application using various GPT models, Claude, Gemini, like all of these models are what's called general purpose pre-trained models.
Sean
Yep.
Andrew
So they're trained on basically the entire internet to get really good at language and understanding language and how things relate to each other, and how to speak and like how what ideas are common like common in language.
Andrew
and then when you are trying to get something from one of these models. There's generally two ways to get the make the response of the models better, to make the models better at solving whatever task you need them to solve. The most common is prompt engineering, of which there there are some sort of guidelines and structure that have been developing, that the model publishers have released and stuff that give you an idea of like philosophically how to to approach prompting models to get what you want from them. And within prompt engineering, there's also, like and I think within the other way as well, there's parameter tuning. So in addition to just like writing good prompts or breaking your prompts up into multiple steps,
Andrew
There's also these parameters that you can tune, temperature, max K, max P, these penalties you can you can provide that'll help create slightly better outputs from from the model. And then there's fine tuning. And fine tuning is taking a pre-trained model and training it further on a smaller, more focused data set. But in general, from what I've seen, I think the best practice is you you want at least a couple thousand examples for fine tuning to be worth it. And fine tuning actually creates a new instance of that pre-trained model that is where all of the like
Andrew
essentially the vector math inside the model that like pout makes the model work, all of the like weights associated with that math have been tweaked. So it sort of it sees similarities oh between words a little bit differently, and that makes it better at certain tasks.
Andrew
Which for a long time, I had heard people talk about fine tuning and training, but I thought training happens like when you were creating a brand new model. It took me a long time to realize that you can like, that fine tuning was training a pre-trained model. and And when you do that, you are actually creating like your own version of chat GPT. And one thing I'm still really fuzzy on is like,
Andrew
partially because I think it's complicated, is just like what a model actually is. like I think it's an application, essentially, running at the GPU level that stores a bunch of these weights and like ah ah vectors and shit in memory is what I think is happening. And then there's like there's like code that goes into creating all of these like weights and stuff. And then like once the model gets loaded into memory, it's like active. And I think that's it's think that's like a semi-accurate basic explanation. And then with both fine tuning and prompt engineering, there's also evals, which is just like
Andrew
When you're testing these things, there's an element of randomness involved. And so to have a a create a really good test, you really need to test things at a little bit of scale. And so rather than just doing one-off tests of like, try this prompt, see what it gives you back, tweak it, see what it gives you back, you really need to be like, let me try this prompt 10 times.
Andrew
And let me compare that to trying this other prompt 10 times. Or let me try to fine tune this model and then run that against a data set and see how the output is. so evals are just a slightly more scalable way of testing models or testing prompts or like methods for getting good outputs from models. So like that was the start of my rabbit hole yesterday.
Sean
Hell yeah.
Andrew
Pause for a breath there.
Andrew
Does that make sense? Do you think I'm like explaining that?
Sean
I mean, you know more about it than I do at this point terms of like, like I, I don't know what a model is. That was the best explanation of a model that I, I just heard.
Andrew
Yeah, it's it it's still really weird to me.
Andrew
that's That's a part that I'm having trouble wrapping my head around fully.
Sean
Maybe maybe someone else will come from Blue Sky and and school us.
Andrew
That would be great.
Sean
Yeah.
Sean
The the fine tuning stuff I did kind of sort of know about mainly because I mean, at least that that sounds like where I mean, someone correct me if I'm wrong on this, but it sounds like what rag is. Have you heard of that?
Andrew
Yeah, so, yeah.
Sean
Trivial augmented generation.
Andrew
I think, rag is still a little fuzzy to me. I think it's connected to fine tuning, but I'm not sure it's the same thing exactly as fine tuning.
Sean
i say i see
Andrew
Yeah, I need to do a little bit more digging there. So what's your understanding of rag?
Sean
It's my understanding of rag is like it's the part of like there's a pre-trained LLM you rag is where you give it a crap ton more data so that it is then but then the output of it is then like it kind of goes through two passes right your input goes to the pre-trained one and then it goes through or at least it gets data from right I don't I don't know I have no idea ah the the ah The thing that kind of made me think about it was when you were talking about how you need like thousands or hundreds or thousands of new, I don't know, entries, new, whatever to to kind of go and fine tune it.
Andrew
Yeah, examples is it the easiest way to think of it.
Sean
So that's kind of what. Word. Yeah, anyway, it's it's just something that I've been seeing a lot, especially just like ads in the ai space.
Andrew
yeah Yeah, rag is becoming a buzzword for sure.
Sean
Mm hmm.
Andrew
I remember reading about it yesterday. That is a piece of of information. I think I was reading about it at the very end of the day. And so that that did not make it solidly into my head.
Sean
Nice.
Andrew
So i'll I'll try to do some research on that and see if I can explain it better next podcast.
Sean
Nice.
Sean
Yeah. I feel like people have been building out courses. Have you seen like the, you only need two weeks.
Andrew
No, what's that?
Sean
It's just people who like have been building courses on like how AI works or how any of it works. the, the, the like phrase I keep hearing is you only need two weeks.
Andrew
to get good at ai
Sean
Just to like understand it, yeah, to like grasp it. don't know how much I don't know how much is in that course, to be honest with you, but it seemed like I got pretty deep into explaining what transformers are and explaining how AI worked and how alums and all that stuff. So.
Andrew
Yet this book did a little bit of diving into what transformers are and like how vector math works and then it gets pretty deep later on into like a lot of the tooling you can use to build
Sean
Mm-hmm.
Andrew
AI-based applications and to test prompts more effectively. like you It gives you code snippets for testing things in Jupyter Notebooks and working with Lang Chain, which is a set of tooling for like programmatically building prompts and using prompt templates and and stuff.
Sean
yeah yeah Nice.
Andrew
Yeah, super fascinating. It's been really fun to so just like hardcore nerd out on this stuff. So one of the things that that has started to become a little bit more clear for me as I've been doing all of this and looking at different tooling and and whatnot, I think I'm starting to see, so there's obviously like lots and lots of tools cropping up to help you
Andrew
build to help AI developers build and work with things. There's like, so like Lang chain has like Lang hub or Lang Smith or something like that, which is like, there's this whole category of tools that's like prompt engineering, like kind of,
Andrew
as a service, not prompt engineering as a service, but it's like prompt engineering infrastructure where you can have like a data scientist. or like a product manager go in and like tweak prompts and test prompts and like deploy prompts. And like you can basically make a ah ah call to one of these tools and it'll route things to like the right prompts or the different models. And and so you can kind of abstract that part away and and create some tooling around it. And they have stuff for like handle analytics and like you know inspecting calls and all that.
Andrew
So there's like all these tools for you know supporting AI developers, but then within the realm of AI tools, and this is what I think is most applicable to like miscreants clients. So many of whom are trying to build tooling, not specifically specifically for AI developers, but for security practitioners. So for like the end user type tools, I think
Andrew
I'm seeing things falling into one of three categories that are kind of on a spectrum of, calling them a spectrum of sophistication isn't quite right.
Andrew
think maybe a spectrum of how autonomous they are is is the right way to look at it. So on one end of the spectrum, you have what I'm currently thinking of as wrappers or tasks.
Andrew
So this is people using AI inside of more traditional interfaces, traditional tools to handle one-off tasks with varying degrees of customization. So like Lex, you've played with f Lex, the writing tool. So like Lex will expose ways for you to you know apply AI to a document editor. And it will expose its prompts and let you tweak the prompts and create new prompts and stuff like that. But but the AI is largely being used in kind of for one off things. It's like, here's a task, do this with the AI. Some people are using the AI
Andrew
behind the scenes to be like all right let me analyze this thing and surface this thing you know let me analyze alerts in your sim and surface like the three most important alerts that the ai has decided are the most important so these are kind of looking at these as like ai ai wrappers maybe isn't quite the right word but like task-based ai systems
Sean
Mm hmm.
Andrew
And then moving a little bit up the autonomy chain, there's workflow builders. So I have a friend who's working on one called Scout OS, and these are kind of like your Zapiers for the AI world. So they pull data from different systems, traditional systems, and then let you create workflows and like pipelines, chains, where you can like pull data from Google Drive based on a trigger or scrape a website or something like that, pass that data to one LLM with a set of prompts that you can figure, get that data back, pass it to another LLM or the same LLM again, and then like pass it to some application or some
Andrew
you know output on your website or something like that. And so I think of these as like workflow builders, the Zapier of the AI world. And then there's agents.
Andrew
And agents, I'm seeing integrated in two ways. Sometimes it's like, hey, here's our tool, and we have our own proprietary agent that we're going to make it easy for you to use. So from what I can tell like Clay,
Andrew
ah ah popular the tool that's getting super popular in the sales prospecting space. Clay's agent is like, they've built it and you can give the agent tasks and it'll go off and do those tasks and like come back with information. And then there's stuff like Lindy, which is an agent builder, it lets you, gives you a no code or low code way to create your own agents. And agents in theory,
Andrew
Like the best agents are relatively autonomous and they achieve that autonomy by pinging back and forth between the real world and an LLM. So they'll, you know, take, get some data from the outside world, pass it to an LLM and then like ask the LLM for what they should do with it. And then act on that, observe the, the,
Andrew
results of the action pass that back to the LLM and it's kind of this like ping ponging back and forth between a system that they're interfacing with and the LLM and using the LLM to actually create their own instructions. And I think that that is kind of how the market is starting to shake out in terms of like big, obviously there's a million subcategories, but like big umbrella categories of like AI tooling for end users, not for developers.
Sean
Yeah, I mean, I agree. and The ah one one of the slides I have now, and we do like a brand strategy a workshop with a client is like, a what exactly is your AI? And it's a it's a it's a bunch of screenshot, there's like, it's a bunch of superhero related things. But like, are like, is your product like the Justice League headquarters? So it's a it's a platform where you manage various sort of superheroes that do things. Is it Robin?
Sean
and AKA a co-pilot, right? Is it, I forget, there's a bunch of other ones, but like, is it like a Batman utility belt? And it's like a bunch of little AI tools.
Andrew
Hmm.
Sean
It's a bunch of, it's just like an easy way to get clients to kind of describe it. Cause a lot of times back in the, like the end of 2023, they would just be like, we're AI powered. And I'm like, I don't know what you mean when you say that.
Andrew
Yeah.
Sean
Like, is your, is like, do I literally co-write with your, like, is your AI in a cursor or is it a, yeah, is it like an underlayer where it surfaces and bubbles up things and it creates
Sean
And yeah, I mean, I totally agree with you. I like the way all this is going because like I really just don't think chat like pure chat is the way to do this.
Sean
When I was in Virginia, I was talking to a friend of mine and about how, like so as I've been working on this prompt to like turn text, hum like turn any sort of writing into like a more human thing, I've been playing around with like
Andrew
Hmm. Hmm.
Sean
tuning my own prompt so that it would, you know, I would tell it like, okay, like write like 60% like Scott Galloway, 40% like he didn't shop or something like that, right?
Andrew
Huh.
Sean
Point being is, itd be be ah my my first thought was like, it'd be really nice if I could just have a form and make a user do like Mad Libs where, and i' not a user, myself.
Sean
Like in if I had a tool where I could put in the, the what I wanted the ad to write about, who I wanted to add it write like, what words to avoid, default words, default phrases to avoid.
Andrew
Mm hmm.
Sean
But then I was like, it'd be really nice if I could have sliders to kind of mess with the weights of all those things.
Andrew
Yeah.
Sean
But even, so to take that, to take that a step further and to bring it back into like the UX of of like AI, to me, it doesn't make sense that I prompt AI through like my own words, and then AI just takes those words and gives me a response.
Sean
also doesn't make sense for me to write it. Click a button that says, fine tune this prompt, like the way Bolt has. Bolt tries its best guess at fine tuning it, and then it, and then and and then I can hit enter and have Bolt then build a product based on its own fine tune prompt. What I want is, I ah give you an initial bit of text, maybe it fine tunes it, maybe maybe it doesn't, but it builds like a kind of like a,
Sean
a like a panel of switches and knobs and things based on my prompt that I can then change the weights of different things in so that it builds me like a better prompt builder.
Andrew
Mm hmm.
Andrew
Yeah.
Sean
or Or it's a prompt builder that gives me a better prompt, and then I can get to what I want more accurately.
Andrew
There are people building that to maybe not exactly that, but like i I guarantee you, there are people building like sort of prompt builders into
Sean
Oh, cool. Sweet.
Sean
I'm sure.
Andrew
usually as a feature of a larger sort of system. And then, of course, there's there's some people who are just like,
Sean
Yeah.
Andrew
doing that and not exposing that it's like prompt building to the user, but they're just sort of giving you the options to configure and then turning that into prompts on the back end.
Sean
Right.
Andrew
So name likes that naming thing that I really like.
Sean
Yeah.
Andrew
that's I guarantee you that's what they're doing.
Sean
Yeah.
Andrew
They're giving you a bunch of options that are essentially Madlib options that then they're most likely plugging into.
Sean
Yeah.
Andrew
prompt template ah of of some sort.
Sean
Yep.
Andrew
Gleam ah has pre-built prompts and like ways to customize the prompts and like One of the things that they have you do is it is kind of that Mad Lib style where it's like, what's your goal?
Andrew
What are the steps? and they encourage you to like break the prompt into multiple steps, which is, you know, a classic piece of advice. and then, but yeah, I think what you're describing is, is taking that even further, which would be cool.
Andrew
yeah.
Sean
I mean it's cool to see all the UX sort of starting to evolve.
Andrew
Yeah, it's all just like emerging. It's still so early.
Sean
yeah
Andrew
And I've seen people recently speculating that we're like hitting the plateaus of what we can do with pre-training these models.
Sean
yeah
Andrew
In part, it sounds like ah ah we're hitting those plateaus because of like hitting plateaus in like the amount of data. that we have like there's it's hard to get more data to push through these things which is fascinating and weird but this some of what i've been seeing lately is people saying like we're kind of hitting a plateau in like how smart the models are getting the next level of innovation is going to happen more at the application layer it's going to be all right take the models and then
Andrew
you know, make them better at tasks through fine tuning and prompt engineering and like all of this stuff and like creating new interfaces and everything, which it'll be interesting to see if that's accurate or not.
Sean
Yeah, separately a different conversation I've had recently about that is so I've been under under the impression that chat GPC is way worse than Claude now at this point, and I think a lot of people are under that impression.
Sean
But a friend of mine who's been more into this sort of stuff, who has access to, I guess, like Azure AI or something like that, um which is what chat GPT runs off of, or like the actual open AI model. The point being was like, I think it's actually public. I don't know. Anyway, the point the point is that like, it's not worse because the LLM is worse. It's worse because the pre-prompting that chat GPT has is way more extensive and worse than Claude.
Andrew
Interesting. interesting
Sean
Yeah, because you know when you go and tell chatgbt to give you illegal bomb-related things, like it's it's happening at the chatgbt layer that it's not giving you the schematics for a bomb versus like the AI model itself, if that makes sense.
Sean
So anyway, I don't know.
Andrew
Yeah.
Sean
I haven't i haven't looked into it that much. but
Andrew
According to, so there's an LLM leaderboard, there's a couple of them.
Sean
them
Andrew
But one of the ones that was in the book was the one from Hugging Face. um And this was fascinating and eye opening to me to look at because according to this,
Andrew
Gemini and chatgpt40, the one released on the 20th, are at the top of the leaderboard. and like So Gemini has several, so it it seems like Google has been doing a good job of catching up to chatgpt with Gemini, which is really interesting.
Andrew
Claude is, you know, farther down, with like three and a half sonnet.
Sean
Mm hmm.
Andrew
you know, it's in like tied for the 10th position, I think.
Sean
Cool.
Andrew
and there's a couple of different rankings and and stuff like that, but. Yeah, pretty, you know, I think this, the arms race of the, you know, general model companies is going to be very beneficial for people building the application layer.
Sean
yeah
Andrew
Because like, sure, those companies are still going to make gobs of money, but there are like Luckily, chat GPT didn't get out far enough ahead to like completely squash the competition.
Andrew
And so you know we're seeing a healthy competitive market that's driving costs down and driving you pretty constant innovation and like quality improvements.
Sean
Yeah. Yeah. Yeah.
Andrew
And then like if you know application builders can leverage that well, yeah I just heard, heard who's Brian's copodcaster?
Sean
I don't know.
Andrew
Not Sal.
Sean
I don't listen to their podcast.
Andrew
Yeah, anyway, I heard this guy on a podcast saying, you you can just constantly sort of swap things out for the newest best models.
Andrew
as an application developer and so you can constantly make your product better, which is kind of cool.
Sean
Yep.
Sean
Yeah, I wonder, I think that it's probably like best practice now when you're building something AI powered, either to be like multi model or to allow easy like swapping of it. um um Or even it'd be cool to kind of have a tool that you can build on top of that sort of did all that for you but
Andrew
Yeah. And that's what I was talking about with like the pro pro ah the prompt engineering hubs, like Lang Smith and some of, there's a couple of others, but there's a few tools that are trying to like be that like you can abstract out the interface between your application and the LLMs so that data scientists and PMs can come in and constantly tweak that stuff without having to push code to the main code base.
Sean
Sweet.
Andrew
Yeah.
Sean
Nice.
Andrew
Also finally understanding like what data scientists do in this world.
Sean
Yeah.
Andrew
It's like for the longest time I was like.
Andrew
You guys keep telling me you're bad at coding. So like, what do you do? And then it's like, oh, they're, you know, let's, you know, they're not building applications. They're writing code to like test things, you know, test prompts and test fine tune models and all this.
Sean
Yeah.
Andrew
And so it's this like weird mix of like stats and math, linear algebra, and then like a little bit of code or sometimes a lot of code.
Sean
Yeah. yeah This podcast became an AI podcast.
Andrew
I mean, I'm building an AI product.
Sean
ah ah True.
Andrew
It was probably somewhat inevitable.
Sean
Fair.
Andrew
But yeah, one of the things that like all of this is leading up to is we're going to have to make a bunch of product decisions about like which bucket do we want to fit into?
Sean
I'm not complaining.
Andrew
like At least for right now, we're definitely not a workflow builder or an agent builder. But then within the like task or like wrapper bucket, we have to decide, you know, do we want to keep the prompts proprietary? Do we want to give you tools to tweak the prompts like you're describing, like kind of Mad Lib style or like with sliders?
Andrew
Do we want to just set the prompts and go, we know what works best, use our shit? Do we want to expose all of that so you can bring your own prompts and like even bring your own API keys? And we're just the tools for like interfacing between LLMs and your SEO data. A lot of product decisions and you know a million different paths we could take.
Sean
Yeah, it is a cool new layer of product management too.
Andrew
Yeah.
Sean
Cool, sweet.
Sean
is When are you releasing this blog post?
Andrew
Uh, hope. Well, I'm hoping to have it finished by the end of the week and, but then probably doesn't make sense to release it next week, uh, because we are approaching the end of the year.
Sean
Nice.
Sean
We are. This is our last pod of the year, probably.
Andrew
Yeah. Yeah, which is wild.
Sean
Yeah.
Andrew
This year is flown by.
Sean
but yeah What You want to do a read?
Andrew
So probably early Jan, early Jan.
Sean
OK. You get the holidays to kind of write and edit.
Andrew
Yep.
Sean
Use LUX.
Sean
Cool.
Andrew
On that note, you want to pivot into some like year in review kind of stuff.
Sean
Yeah, yeah. Yeah, let's do it. I don't know. How do you want to start it? How do you want to?
Andrew
Do you, do you ever do any year in review type stuff or any like yearly goal setting? Well, we've talked about this a little bit. You're not a big goal setter. You're a, what is it? A fear setter.
Sean
I'm a fear setter. Yeah, I did use some fear setting. Yeah.
Andrew
By the way, Greg told me, our friend Greg that, uh, you completely misrepresented fear setting and like you missed half of the important stuff. So we got to get Greg to come on the podcast at some point and like school us on what fear setting is supposed to be.
Sean
Nice.
Sean
well uh okay anecdote sidebar really quickly i told greg i've told greg this from the past i think i told you that before oh so first of all it works for me okay greg go go over to come on and tell us how to do it and i could do it in a way that angie stops saying it's toxic but one of the things that you know one of my fears is greg one day telling me asking me not just greg asking me what's the progress on this thing and me being embarrassed by it and The reason that's one of my fears is because it literally happened to me when I was in, like, I was gonna say high school, when I was in college, I met Greg at Shmukon, we were chatting about things when he was at Grey Noise, and then I was like, here's all the like stupid like ah ah things I would like to work on and all that stuff, and he was like, what's the progress on ah on that? Out of just complete like genuine curiosity and not like ah not a sort of thing, and I was like, fuck.
Sean
zero zero these are all pipe dreams I'm not executed in any of it so yeah I think that fear motivates me to go do things and execute on things I don't get caught with my like proverbial pants down so yeah i'm I'm kind of curious what what yeah yeah
Andrew
I think that sounds like a miserable way to live, but if it works for you,
Andrew
what Greg says about, about fear setting.
Sean
kind of
Sean
Yeah, I'll just call him after.
Andrew
Yeah. okay. So I think, so I can share like. my 2024 goals that I set at the beginning of the year and how I've done on those.
Sean
Yeah.
Andrew
And then I can I'll try to probably think through like big milestones or any kind of lessons or takeaways from this year.
Andrew
i think I think that probably would be the best format for me. then you want to just tell me all the shit you're afraid of?
Sean
No, I'm not. gonna I'm not going to put that on on air. The dark.
Andrew
Yeah, what do you what do you want to do?
Sean
I mean. you usually by the end of the year I'm sort of reflecting back on what December looked like versus what January looked like and sort of it's very like generally what are my emotions right now versus previously um and that's generally the reflecting I've done I do some year in the review stuff like I kind of I kind of sit and write it's just like always a half written blog post
Andrew
I feel that.
Sean
Yeah, usually, and if anything, the year in review is more like ah like what I'm looking forward to in January slash next year sort of thing.
Sean
So yeah, I can go through and can go through that stuff.
Andrew
Cool.
Sean
What are your goals?
Andrew
Okay, so at the beginning of the year, my goals, and they these shifted a little bit. I like tweaked them as I went and like figured out what I felt like would be more applicable or like be a better goal. So the the very beginning of the year,
Andrew
And I try to always have like a couple goals in like each area of life. So business, relationship, travel slash like personal fulfillment, health.
Andrew
personal content kind of stuff. So start off with get a new business to ramen profitable. Depending on how you look at this, I could maybe argue that it's been successful exclusively because of my freelancing with miscreants.
Sean
Hell yeah.
Andrew
My freelancing with miscreants, which does run through disco shrimp company,
Sean
Doesn't count. Doesn't count.
Andrew
is definitely ramen profitable but yeah neither neither of the two businesses that I product businesses that I started this year are at ramen profitability yet so church use is languishing in you know It's the the forgotten stepchild languishing over on the side. Haven't done any updates to it. It's not making any money. I should probably check to make sure it's even still working. ah And then yeah Meta Monsters going well, but yeah know things always take longer than than you think they will. so we we haven't gotten to the point where we're able to charge customers yet, probably looking at some time in February.
Andrew
When, when we get there. So, that one we're gonna call a miss. Personal one, wanted to go to couples counseling for the first time this year. I'm a big proponent in therapy.
Sean
Nice.
Andrew
of therapy and, you know, Maddie and I, our relationship is getting serious. And so we wanted to like, you know, just get a third party opinion to help us work through some classic relationship stuff and, you know, help us prepare for like the future. Felt like that would be a really good thing to do. Tried that out. Found someone who probably wasn't a great fit, so may revisit this again next year.
Andrew
On the personal like fulfillment side, two that I had were to go to Mexico for a week, wanting to both practice my Spanish and do some traveling in a new country, then go on a week-long backpacking trip. so Those are you know kind of check mark goals, which aren't always the best.
Andrew
but
Sean
Mm hmm.
Andrew
yeah they they worked for me they gave me like a concrete thing i wanted to do i did both of those they were two of the highlights of my year so that was really cool and i'm i'm glad i set those and then i started off the year with three goals on the like sort of personal health and fulfillment side of things so 10 push-ups, 10 sit-ups a day, 10 minutes of writing a day, and 10 minutes of meditation a day were what I started with. And I was doing those for a while, especially the 10 push-ups and 10 sit-ups. The other two were a little bit harder. But they just didn't feel like they were really hitting on what I wanted. like I didn't feel like 10 push-ups and 10 sit-ups a day was enough for me to like really make any progress, being healthier or fitter.
Andrew
And so I ended up deciding to run a half marathon instead, which was a really good thing for me. got me yeah being active and like working out consistently again. And then I ended up shifting the skull towards the end of the year to just like, I want to have a routine of working out three times per week.
Andrew
Which I am now doing pretty consistently, thanks um yeah the half marathon was a good starting point, it was a good first step and then after the half. signed up for the service called train well, which is like a virtual personal trainer. And so I have a real person who sets.
Andrew
a workout for me in the app week. And then I go to the gym and I follow that workout and I text him about how it's going track it into the app. And that's been really, really, really positive for me. And then I sort of scratched the other two and replaced them with 12 new blog posts because I realized like really what I wanted from the writing was I just want to be writing on my personal site more consistently.
Andrew
and Again, like the goal was really personal sight. I have written a little bit for MetaMonster, but i I don't think I'm going to hit this or come anywhere close to it. I think I'm going to, I've got six written so far, seven if you count one of the ones for MetaMonster. So into the year, I'll be somewhere between eight and nine.
Sean
But your your blog posts are like, lengthy, right?
Andrew
Yeah, they are.
Sean
Your blog posts are like, yeah, short chapter, yeah.
Andrew
Yeah.
Sean
Yeah.
Andrew
Yeah, some of the ones I got posted this year were like some of the longest I've ever written.
Sean
Yeah, your blog posts get like copy edited by your friends, including me, but I was always like, whoa.
Andrew
I think the the Camino blog post, the final one was like 6000 something words, maybe, maybe even longer. So. Yeah, they are. It is a lot of work to to do that, but still want to continue to try to build that writing habit. Yeah. So that's that's kind of how goals went.
Andrew
Yeah. any Any thoughts or we want to jump over to to kind of your stuff and then
Sean
what are your what are your goals for next year do you have them them set yet or
Andrew
I haven't set them yet. Get a new business to ramen profitable will probably carry over. Would love to to see that happen with MetaMonster.
Andrew
I think I'm going to plan, I think
Sean
What are your goals for next year?
Andrew
I want to do more backpacking. So this year I had the goal of going on a week long backpacking trip, which I did.
Sean
Do you have them set yet?
Andrew
And it was an awesome experience. I, I went with my dad and we through hike to the foothills trail, which was 76 miles through North and South Carolina.
Andrew
really, really cool experience, but I think. That's basically the only backpacking trip I did this year. And I know that I love backpacking. I'm super happy when I'm doing it. And so I think next year our goal is going to be like either five or six backpacking trips.
Andrew
Got to figure out my like Spanish language travel goal. Haven't fully decided on that yet.
Andrew
Yeah, got to figure that out. Uh, we'll probably stick with working out three times per week. that's been really solid. I might end up tweaking that as I get into the year with something more specific as far as like a performance goal. But I think really just, if I can stay consistent with that through the year, that would be great. I'll probably have another writing goal again. You know, I might just roll with 12 new blog posts again and see if I can get it done.
Andrew
And then I might bring back the meditation thing because I think that that is something I've never been able to stick with it for more than a few months. It's always been super hard, but I know how big a difference it makes on my mental health when I do it. So.
Andrew
Yeah, probably something. Goals for next year will probably end up looking pretty similar, honestly. I feel like I'm pretty happy overall with how this year went. like I love the work that I'm getting to do. I've enjoyed the work I'm doing through miscreants and love the freedom and flexibility I have right now. I'm really enjoying working on MetaMonster. I'm stoked to be working with Austin again.
Andrew
you know I want to make sure I maintain some balance and I do still travel and you know take advantage of the flexibility I have and not get so focused on like building for the future that I don't live in the moment now.
Andrew
And then, yeah, always trying to spend more time outdoors and trying to build that writing muscle. so
Sean
Nice.
Andrew
Probably pretty similar. I feel like I think this is the happiest year I've ever had.
Sean
Whoa. Damn.
Andrew
I think that 3031 transition year for me, I think I am the happiest I've ever been. I feel the most like myself. I feel like, you know, I've I feel like I finally know how to dress well and like, I've been getting tattoos and I love my tattoos and I just, I feel like, yeah, I feel like the most me version of me and I've, I've, have felt on the whole, very happy this year.
Sean
Uh-huh, debatable. Sorry. Sorry. Uh-huh.
Sean
Well, I'm happy to hear that. That's awesome. I would probably, I'd probably say same. yeah, I do feel like every year I think that though, you know, I, um, which I think is a good thing, but go ahead.
Andrew
Fuck yeah.
Andrew
I don't know that I thought that last year.
Sean
Fair.
Andrew
I think parts of last year I was extremely happy, but parts of last year I was extremely stressed.
Sean
Mm-hmm. I'm, yeah. Yeah, I remember.
Andrew
And I think I've had a handful of of years running Crit that felt that way as well. yeah like
Andrew
Crit and then personal stuff that's ebbed and flowed over the years, like
Sean
Hmm.
Andrew
There have definitely been years where even if I was proud of where I was, I wasn't, I wouldn't have said that.
Sean
Hmm. I see. I see. I see.
Andrew
Yeah.
Sean
I think, yeah. Well. okay well that's good i'm happy to hear that um um yeah thanks for letting me be a part of it yeah yeah you could still work on the dressing part but you know that's okay that's a 30 that's a 3032 thing um um yeah yeah when you have like wider pants and and what okay
Andrew
Fuck yeah.
Andrew
i Dude, I bought a pair of giant chinos. Yeah, I haven't worn them yet, but I did buy them.
Sean
okay I see.
Andrew
So I'm trying. I'm trying to widen the pants.
Sean
and Okay, I'm curious how um how um curious how they
Andrew
Yeah, I'll send you a pic.
Sean
okay sweet and see so yeah The most recent pic I have of you is still you in the Louise costume. so
Andrew
That's hilarious.
Sean
cool cool I think that's always interesting. like now When I was thinking about and hearing you go through your goals, the thing that's interesting to me about goal setting is that you set it at the beginning of the year.
Sean
and like I like come up with random goals throughout the year. like a goal like not They're not goals.
Andrew
Mm hmm.
Sean
they're not like measurable or actionable sort of thing they're just kind of like I would like x y z and then I go and achieve it and it kind of just you know I don't set a destination necessarily by the end it's just kind of moving across
Andrew
Hmm.
Sean
So yeah, it's always sort of a roll of the dice. Yeah, I also think it's been it's been a good year. I think I'm proud of where I am. I think it is the hardest. I think every single year when I look back at it though, it is like the hardest I've ever worked in my life. And especially leading up to probably even the beginning of this month or last month, I think I'm finally, I think when I went to the conference and when I was in Virginia and like the beginning of this week,
Sean
Maybe it's because client things are spinning down. Maybe it's because we just launched like six different websites in the past two weeks. But it feels like there's a little bit more breathing room.
Sean
There's at least enough breathing room to you know work on a very tiny blog post every single week. I'm trying to keep that habit up.
Sean
Thank you for being of the one person that clicks and reads it.
Andrew
Yeah.
Andrew
I've loved them so far. Super fun.
Sean
Thanks. Thanks. Thanks. They're all just stream of consciousness. Yeah. I think I'm I'm very well, okay. I think I'm looking forward to next year. I think last 2023 was pretty rough comparatively. I think i think last year in december the feeling was holy shit we survived another year and it's pretty disappointing because like in accruals we were over you know we were over seven figures and not accruals and it was dicey i think this is the best year mr insta has ever done as well partially also thanks to you so i appreciate that
Sean
think like, as an agency, we do have goals, or I have goals for the agency, which is, I finally actually started sitting down and writing out like a website SOP, mainly maybe also because you asked, like what our process was or she may ask and I was like,
Sean
Is it good that Greg listens to these things?
Andrew
It's fine.
Sean
No, we we we always have like like I have a process in my head, but it it hasn't been communicated. And you know the team now looks so different than the team two years ago. i been One of the main things that's been on my mind is this whole whole idea of like batch delegation and like fully delegating the process so someone else owns the process rather than I still own it.
Sean
I still do the first 10%, the last 10%, and everyone else kind of does the middle. So in a way, one could say I'm trying to get out of founder mode um and trying to be like an owner of a company rather than.
Andrew
Hmm.
Sean
of the things that, of the ideas I got from JJ yesterday on a call was this idea that like at large consultancies or firms there's like partners and then there's like people, right? And partners sort of fly in and they kind of, they meet with the client, they do like, they they kind of like say smart things and they set the general direction of where a project is gonna go.
Sean
because they have, their value to the firm is the info in their brain, not them doing the actual work about something. So I think collectively there's a goal to kind of put me in that spot. I think as an agency, we've proven out the fact that we can do some really great work and like like groundbreaking, like industry leading work. So that's pretty cool with some of our clients.
Sean
Yeah. But yeah, I think if anything, the theme of this year has definitely been like a deterioration of habits.
Sean
I stopped going to the gym, stopped having a good sleep schedule, stopped having like any sort of good healthy things. So maybe 2025 will be the rebound year.
Sean
oh Yeah.
Andrew
Yeah, I had a lot of that, I think in 2020.
Andrew
uh, three, which is what led to a lot of those being goals for this year.
Sean
Mm-hmm.
Andrew
It's like, okay, it's time to fix this.
Sean
That makes sense. Nice. Nice.
Andrew
and you know, I had, I got fired and then had the, the Camino and all my time off to like kind of assess, which was nice. Um, I think it's often hard when it's just business as usual to kind of recognize, Oh,
Andrew
this is I've lost those habits and I want those back and like to make start making the shifts to like get those back can be really tough.
Sean
yeah
Andrew
Yeah. Yeah, I think personally. A goal for you should be. To have like, I don't know exactly what it looks like, but, you know,
Andrew
To have a like some number of website projects by the end of the year.
Sean
Yeah.
Andrew
That are executed entirely without you.
Sean
Yeah. Cool. Mm. For sure.
Andrew
Yeah, like you come in you close it you're there for the kickoff call and then you're gone. And like. you know, check in from time to time, sure, but like, you are not involved in the day to day of like a ah branding and website project. And I think like, ideally, by the end of the year, that's starting to become the norm. So I don't know what that looks like, in terms of goals, goal setting exactly, but like, you know, maybe it's
Andrew
You know, maybe it's by December, I'm not involved in any of the at current active projects. or maybe it's like, I'm not involved in 50% or I'm not involved in one. There's one that's running entirely without me.
Andrew
Yeah, I don't know, but I think that would be a good something along those lines would be a really good goal for for you based on what you've told me about what you want from the business.
Sean
cool. I think that, yeah, it's a pretty good goal to have. I don't think I thought about it that way. It's and so annoying when these things feel so obvious in retrospect, and and but someone else has to be the one to say it, but yeah. The other the other one I was working with was more like, yeah, i think that's ah I think that's a good, I think I find myself always thinking in like a more narrow field of vision, right?
Andrew
Mm hmm.
Sean
I think my thinking was like abstractly, I would never log into Webflow by the end of the year, or I wouldn't have to log into Webflow by the end of the year anymore, but I think what you're saying is a more concrete, specific way to, yeah.
Sean
Yeah, that's good, that's solid. I think when you say that, I also remember that this year has definitely been like the year of humility like of of like ego death and humble being humbled mainly because there's like a fundamental belief that i can do like i'm the only person that can do a certain part and i've i think as we have budget and i've met more people and i've worked with others and even clients like i've worked with some really great product marketers this year and and have met also like good product managers who i think would have made good product marketers
Sean
the like The general thinking is like, like oh, like there there exists people. There's at least one other person in the world that can do this thing, which means that I am not the only person who can do this, and therefore that but means I can hi probably hire for this person this type of person. so anyway not feeling special I think has been a good like relief on it a little bit yeah because the sales pitches that were special right this whole sales pitches yeah
Andrew
Hmm.
Andrew
Do you have any personal goals? Anything you want to accomplish outside of work?
Sean
I have like one main goal and it is and and this is like a general life goal like this is like a one main thing that I would never like to experience again and it's that I would never like to be in an Airbnb on PTO in Korea or Japan and and hammering away at work and like watching the day go by and yeah but in terms of like
Andrew
Mm hmm.
Andrew
Yeah.
Andrew
Taking, taking like a week long trip without where you are completely unplugged. Like you delete Slack.
Sean
I didn't say that.
Andrew
I think that I i think that you should say that it is doable.
Sean
I didn't say that. Uh huh.
Sean
Gotcha. Fair. Fair enough.
Andrew
It is a hundred percent doable.
Sean
Yeah. Yeah.
Andrew
You just have to trust the people around you to execute on things and like not be there as a backup.
Sean
Well, we'll slate it for Well, and think it's a good I think it's a good eventual goal to have.
Andrew
Yeah.
Sean
I think if i yeah it'd be nice to pull myself out of the agency for a week.
Andrew
Cool.
Sean
But yeah. yeah
Andrew
You gonna, do you have any, like you mentioned like the gym and stuff like that. Is there anything like that that you want to start doing again? You gonna, you want to join me in my get swole journey.
Sean
No. and
Sean
Yes, but not like in your train well personal trainer journey.
Andrew
That's fine. Do whatever works for you.
Sean
um
Andrew
I don't care.
Andrew
Sick.
Andrew
Dang.
Sean
I have no idea. I have no idea what my personal goals are. I feel like I just, as of the beginning of this week, got my head above water, so this is like the first time I'm like scanning the personal life horizon to see what's up.
Sean
So yeah, and so
Sean
I think I've, I think I have general like writing goals and or habits I'd like to establish and that's pretty much it. Yeah, I'm working on a blog post though. I don't know if it's gonna get finished. Wrap up on some funny thing.
Andrew
Mm hmm.
Sean
There is a specific, it it is it is basically about the fact that like and like there is a specific type of business, a class of business when I see it.
Andrew
OK.
Sean
I'm always super jealous that somebody gets to do this. um Because it's always it's always like kind of quirky, kind of specific, and from what I can tell, it makes a bunch of money.
Sean
And it's just like one thing. So for example, alarmcontrols dot.com. Right.
Andrew
Okay.
Sean
Although I think they're owned by a conglomerate now and it's not the best example. But when I found out about them, my understanding was that they just made custom alarm controls like, but or not alarm controls, maybe it's alarm.
Andrew
For high traffic and high security applications, this
Sean
yeah yeah yeah they make when i found out about these guys uh it was through the fact that they made like specific like key switches for uh like push buttons and stuff for for alarms another good example of this is like edgardens.com right or a king flip chart king flip chart just makes like the flip chart that if you see like to tony robins or alex from ozi like the ones they use on stage like
Andrew
Mm hmm. Mm hmm.
Sean
One schema is a good digital one. Stage timer is probably one that you know of as well.
Andrew
Yep, no stage timing.
Sean
They're non-AI. They're non-pretentious. They're just like a thing.
Andrew
Uh huh.
Sean
And they're not like, from what I understand, they're not like, they're produced in a factory owned by the company. They're not like a drop shit e-commerce sort of thing.
Sean
They're just like a thing that they do.
Andrew
One of my best friends almost started a dumpster company this year, just like manufacturing dumpsters.
Sean
amazing amazing yeah yeah that's that's incredible that's uh um um oh it's just it was just like i thought it was cool so i so i was running it halfway i kind of lost steam i'm gonna be honest with you halfway through it was just me kind of like
Andrew
And I was like, that's so cool. That's, that is so AI proof. I love it.
Andrew
So what's the blog post?
Sean
like in of these companies and trying to dissect why I liked these companies. Yeah, may never get published. I may just stay as a draft.
Sean
I've been doing this a lot where I like write half a thing and then it stays as a draft because I lose steam. So I haven't published much.
Andrew
Hmm.
Sean
I'm taking it more as I'm not publishing it because it doesn't feel like the right fit of things I would like to write.
Andrew
Cool.
Sean
So that's why we're being killed.
Andrew
Fair. Yeah, that's fair. Cool. Cool, man. Well, here's to, you know, a good 2024 and an even better 2025.
Sean
twenty twenty five
Sean
yeah I hope you are happier next year than you were this year.
Andrew
Hey man, me too. um Yeah, sounds great.
Sean
I'll see you later.
Andrew
Go man. Bye.
Sean
Bye.