[00:00:00] Phil Gamache: What's up everyone, today we have the pleasure of sitting down with the acclaimed Brittany Muller, founder and consultant at DataSci101 and former SEO scientist at Moz. Brittany started her career when she moved to Breckenridge, Colorado, chasing fresh snow and snowboard hills. She connected with a local realtor who introduced her to SEO.

[00:00:23] And after discovering search data, she never really looked back. She spent seven months preparing to rank her personal site for the term Burton U. S. Open and ended up ranking ahead of Burton. com and received a call from their marketing team who invited her out to dinner. And this spurred her to start her own agency, which she ran for several successful years.

[00:00:43] But after being on the cutting edge of SEO and doing the speaking circuit at conferences around the world, Brittany started getting hungry for a new challenge, enter machine learning. She stumbled upon Harvard's Data Science 109 course while searching GitHub repos and dived super deep into this [00:01:00] new field.

[00:01:00] She was eventually poached by Moz, where she spent four years as senior SEO scientist, where she rewrote the Beginner's Guide to SEO amongst a bunch of other content and continued her SEO research. And she later joined Hugging Face, the fastest growing machine learning community and open source ML platform.

[00:01:16] And today, Brittany has returned to her entrepreneurial roots as machine learning and SEO consultant and the founder of DataSci 101 with the goal of making LLMs like Chad GPT as accessible as possible. Brittany, it's an honor to have you on the show. Thank you so much for your time. 

[00:01:32] Britney Muller: Thanks for having me, Phil.

[00:01:33] That's quite the intro. I'm so impressed. The depth of your research. Incredible. Incredible. 

[00:01:41] Phil Gamache: Thank you so much. Yeah, pride myself on coming up with some interesting questions. Like, I, I, most of the time, the most interesting speakers to chat with are folks that have done the interview. Circuits and I like find interesting answers and then I can just double down on like, okay, like, let's, let's like drill down on that one point that you did [00:02:00] there.

[00:02:00] So there's no shortage of interviews with Brittany, especially SEO context online. So yeah, it was, uh, but on top of like the, the LLM stuff that you've written already. So yeah, super excited to chat with you today. I want to start by just like talking about how. He was drunk on AI hype way before a lot of us, especially in marketing, dating almost back to like 10 years.

[00:02:24] When you, you first built your LLM as an SEO Rap generator combining Moz, uh, Rand's, Moz blogs with Beyonce's Lemonade lyrics. It's safe to say that you were almost like a decade ahead of most when you shifted your focus from SEO. To machine learning, what was the triggering factor back then that led you to have this crystal ball moment and go down this new endeavor?

[00:02:48] Britney Muller: That's a good question, Phil. Oh my gosh. I think, you know, we're all so different as people and I think unique things fuel unique individuals. [00:03:00] And for me, it's always been, like, this passion and, like, love of learning, love of tinkering and understanding how the hell something worked. And I had that for a really long time in SEO, and I still get to experience little, like, spurts of that.

[00:03:14] But I needed something to really sink my teeth into at that point, because I had sort of plateaued as far as SEO experiments and all that stuff goes. And when I learned about machine learning through that Harvard CS109 course, I, my life, like again, like I knew it was forever changed by this because Of the innate power of feeding a model data and the letting it learn patterns and representations on its own.

[00:03:46] I mean, the examples that we were playing with throughout that class were already so powerful that I just, I was so confident and so sure that this would be part of the future that I knew I couldn't let go. And I [00:04:00] had to like continue deep diving. So I remember. You know, when TensorFlow first came out and how excited I was to write, like, it was over 200, probably close to 300 lines of code to do like a very basic kind of linear regression model with TensorFlow.

[00:04:17] And now that's all been abstracted away into just a couple lines of code. So it's exciting to see how this has progressed. And I'm sort of a Frankenstein developer of sorts. And so I like to find something. That has already been built to do something and then I like to sit there and consider how can I break that and make it do something similar, but for this over here, um, and so, yeah, I just started incorporating different models into my life, into my work, making really silly things like I remember in Denver, I had a co working space and I used to uh, Walk my computer around like a baby when I [00:05:00] had built my first MNIST model because it was, it could recognize handwritten numbers with like, a success rate of something like 90 something.

[00:05:09] I was super proud of it. So I was like, write down a number, write down a number. Like, look at that. 

[00:05:13] Phil Gamache: I was 

[00:05:13] Britney Muller: so proud. Um, and just, yeah. I'm obsessed with it. So fun. 

[00:05:19] Phil Gamache: Very cool. What makes you amazing, I feel like is like the, the curiosity that spurs you down this new research path, but also the time you take to educate peers and like the rest of us in the marketing and the SEO community.

[00:05:34] And I feel like you're in Denver, your endeavor with DataSci 101 is amazing. Right in, in line with these, with these goals, like you established the, the new venture to educate the public about the creation and the capabilities of LLMs. And your, your aim is kind of to foster like public discourse, educate people, but also interesting ideas like alleviating the irrational fears around [00:06:00] AI and ML and cultivate more trust and support, just like a more considered approach to, to, to the future of it.

[00:06:07] Right. Right. And so. You've actually already released some content. Uh, I've, I've had the pleasure of reading part one and two of your guide on, on LLM and you're kind of on tour right now, like speaking at a bunch of different events. Uh, I watched your, uh, recorded talk, uh, that you did for, for Brighton SEO.

[00:06:24] What's next for Brittany and Datasite 101? Like what do you have in store? 

[00:06:28] Britney Muller: Yeah. So I feel like, first of all, I just love finding really cool stuff and being like, Hey, check this out. Look at what this can do. Like you can do this for whatever you're working on. And so it's just, that part is so fun for me.

[00:06:42] And I think as marketers, we have been absolutely robbed with statistical knowledge. Like it never in, you know, my marketing career path. Was anyone ever like, Hey, like you should really [00:07:00] get a lot better at statistics cause you're going to be dealing with data all the damn time. And I wish someone would have like, you know, suggested that or helped me with that.

[00:07:11] And I think just even as like a general, like anyone, you know, that consumes content, it's so important to have just basic statistical understanding. But truly from the marketer perspective, I was like, When I took a step back and the more I had to like get statistical and data science knowledge for some of the machine learning work that I was doing, I realized like the gaping hole in the marketing industry that, you know, really is the sort of data centric data literacy gap that would empower so many marketers.

[00:07:48] To make more strategic decisions, um, to find better insights to surface, you know, to surface some of those things easier. And so that has been a [00:08:00] real passion project of mine. And I have been working very hard on different content mediums to, um, to communicate some of those things and communicate it in like a very different and fun way.

[00:08:12] I think the long form blog posts, like. is, is great for me because I, you know, that's kind of traditionally how I've formatted different pieces, but I love to take a page out of my friend, Daisy Quaker's book, where she's always like, all right, you build this big long resource. Like you have the turkey, make turkey sandwiches, you know, like go make sandwiches with it, Brit.

[00:08:35] I'm like, Oh, we all should be making more sandwiches. So that's, I'm working on that. 

[00:08:43] Phil Gamache: Very cool. Yeah, I definitely agree. It's, it's been the, the, the so far part one and two, like insanely valuable and approachable, but I love how beautifully written it is and filled with funny, but like relatable analogies as well.

[00:08:58] Like my, my favorite quote [00:09:00] is really early in your first part, part one. And it's, uh, LLMs are essentially aliens from a different universe. While they have access to all of our world's texts, they lack genuine comprehension of languages, nuances of our realities, the intricacies of human experience, and knowledge.

[00:09:15] And I didn't really have a good question to ask you about this. In all honesty, I just wanted to create a mid journey image with the aliens to accompany this header. But I'd love for you to unpack this a bit further. 

[00:09:28] Britney Muller: Yeah. Yeah. That's so funny. So I remember reading like, and one of the more technical books that I have, I forget which one it is, but, um, they're describing it as like, you know, these large language models are essentially, it's essentially feeding some being in a black cave, all of the world's text and we have no understanding of how this thing lives or, you know, we have no shared experiences.

[00:09:58] With these things. And so [00:10:00] that's what gave me the idea for the alien. Um, it's so funny and I've worked so hard to come up with these analogies. I find little post its all over. Like yesterday I found the one where I was like, Oh, baseball, like baseball things to communicate, encode or decode. Or like, it was silly, but I think it's important to kind of break that stuff down in a way that people can connect with because it is really hard to communicate some of this stuff.

[00:10:24] And, um, yeah. Yeah, if that makes sense. It really is essentially an alien in a dark, black cave that has consumed all of the world's text and is just really clever at sounding proficient at a bunch of general things, but when it comes to like going deep on one particular subject or any sort of edge cases, it, I mean, has no idea.

[00:10:47] And then you take it to like the real world where, you know, even us doing this podcast right now, we might assign like. a large language model agent to say, Hey, go do this [00:11:00] podcast with Phil, but it doesn't understand, like, take one step with your right foot once, like all the things we don't even think about, like breathing and blinking.

[00:11:10] It doesn't know anything about right. It's crazy. It's actually crazy to think about. 

[00:11:15] Phil Gamache: Yeah, it's really good at making it seem like it knows what it's talking about, or it understands things. And I think that like, Most people would agree that LLMs aren't knowledge databases, but there's like a bit of misalignment on whether they're reasoning engines or not.

[00:11:33] Uh, you've explained in a few spots that chat, GPT is a giant probabilistic engine without reasoning capabilities. Uh, but one of the articles I've read, uh, the CEO and founder of Every Dan Shipper, he wrote this article, which he claims. Uh, that after attending a talk at Sequoia where Sam Altman said that GPT models are actually reasoning engines, not knowledge databases.

[00:11:56] So, interestingly, when I asked ChatGPT, like, [00:12:00] he sides, or it sides with you that it can stimulate reasoning like responses, but is not a reasoning engine. So, would love for you to explain that stance further. 

[00:12:10] Britney Muller: Oh my gosh, this is so wild and probably gets a bit convoluted, but I love this question so much because I think it's an important thing to consider, especially because so many people are really hyping up this technology to be these like all knowing, all powerful things.

[00:12:29] Phil Gamache: Yeah. 

[00:12:30] Britney Muller: Um, and so it really does kind of come down to. What, what is reasoning, right? Like, do we even have a clear definition of what reasoning is? And if we do, does this fit, right? Cause if we were to say like, just sounds confident about, you know, various domain questions, like sure, then it might be reasoning, but.

[00:12:54] It's a, it's still a word guessing machine, and I think we really [00:13:00] lack as an industry, just core definitions of some of the words we sling around carelessly as marketing jargon. And I think that's really dangerous because it also starts to paint this technology as like perfect or, you know, human like capabilities, which it certainly does not have right now.

[00:13:19] Um, and we're also talking in just the purest form sense, right? Like a large language model is inherently a word predicting machine. Um, but if you connect it to different things, right? Like action or different, um, informational resources, it can start to do some, some sort of like rough planning. But even then it's like, is it really?

[00:13:43] Um, and here's, here's where I like, I really love the question and I'm continuously challenging myself to think differently. Because I, I, to be fair, like, I go back and forth on a lot of these topics, and I really look towards the experts and [00:14:00] the people developing this technology to have some of these answers for me.

[00:14:04] And so this past, um, the past week I was at NeurIth in New Orleans, which, if you've read any, like, computer, Development books or machine learning books. It's hard to come across one that doesn't mention NeurIPS as like a really like founding part of the space and the competitions it's hosted and developed and blah, blah, blah.

[00:14:24] So really it was excited to figure some of this stuff out there. And there, Phil, there was a whole track one day around LLM reasoning. And let me tell you, that room was so packed. It was, it was the biggest fire hazard I've ever, ever seen. Like we were shoulder to shoulder, like all along the sides, there was.

[00:14:47] a couple thousand people in a room that should have only had like, it was insane. And I remember just standing there like, what the fuck? Like, this is crazy. Because again, like, inherently, [00:15:00] so many experts are very clear that like, these are not reasoning engines, but I'm standing there like, what are we all doing here?

[00:15:07] Like, what, what are we? Um, and then, yeah, I used to be some of like the founders of computer science in different talks of, like I was sitting in one, um, that was about LLM agent collusion and how agents can like team up and like gang up on like the user. And I mean, it's just like, we got into like crazy world there.

[00:15:29] And so it was interesting to see such conflicting thoughts and ideas throughout the whole conference. Right, you have all these experts saying like, they're not reasoning, then you have these tracks that are all about reasoning, and people are everywhere, I mean, so, again, I think it's important to keep an open mind, but I think we do start to get a bit carried away with ourselves when we don't have a clear definition.

[00:15:54] So it really comes down to that. And I hate the benchmarks we currently use for LLMs. There's a bunch of great [00:16:00] resources around this in particular where like human exams are not a good benchmark for large language models because they're wrong. They've seen iterations of them. Like they're just parroting back answers to that stuff.

[00:16:14] So that's a terrible, terrible, um, benchmark. And when you, Um, change the questions in a way that they haven't seen before. They do very poorly. Um, another resources, uh, Emily Bender at all wrote, uh, this really great paper comparing these large language models and benchmarks to Grover and the everything museum on Sesame street, where it's like these arbitrary rooms, right, of like one room is things that are on walls and another is like things that like water, like it's just so random.

[00:16:49] And it really proves how impossible it is to essentially. Evaluate a system that is supposed to do [00:17:00] everything, all, you know, this general knowing system. We just, we don't have anything like that yet. So yeah, it's a, it's a, we're living in a wild time, Phil, what a time to be alive. 

[00:17:14] Phil Gamache: I feel like I, I sparked, uh, something that could probably turn into its own, its own podcast episode in a bit.

[00:17:20] So. I wanna, I wanna like, what, what was your, your main takeaway that you walked away from, from, uh, from that, that fire hazard of a, of a talk? 

[00:17:30] Britney Muller: Oh my gosh. Yeah, I walked away with a, I, you know what the biggest takeaway was from like the conference as a whole? Is like, this shit is so fuzzy. No one knows!

[00:17:41] That's like the big takeaway, is like, I saw so much, like, conflicting, information and even research that it was like, wow, I really thought I was going to walk away with some concrete, like, oh, this is what I'm going to take into 2024 for like, you know, [00:18:00] talks and work that I do. And instead I'm like more confused than I was before.

[00:18:06] Which is crazy. Um, and it was also just really cool though, to also see, like, these are humans, like a lot, a lot of the people we kind of rip apart online and like seeing, seeing them in person was like, wow, like we're all just trying to do our best and, you know, and unfortunately some of these bigger tech companies, like what's in their financial best interest, isn't always in the best interest of the general public and that's something I'm constantly like keeping an eye on.

[00:18:37] Phil Gamache: Yeah, as much excitement as there is around this new technology, there's a lot of considerations and maybe like the darker side of things that these big tech companies aren't shining too bright of a light on. And I feel like you're doing a good job at demystifying that and disclosing that to the public and a lot of your talks and In part two of your [00:19:00] guide, you offer another compelling analogy.

[00:19:03] Uh, LLMs are like hot dogs. Understanding it's complex and sometimes unsavory composition is essential before deciding to take a bite of that hot dog. But many often deliberately ignore what is inside that hot dog may mirroring this idea of like, what, what you're talking about in, in part two. So like.

[00:19:25] The, the key question now is like, how, how do we get more people to care about what's in their hot dog or how the LLM is built? How do we engage more people in this conversation about the ethical and the technical aspects of AI development? 

[00:19:41] Britney Muller: Oh my God, I love that question so much and I've been joking and I really might do this now, like, because it just, I keep saying this, but I want to make t shirts that just says what's in the data because that's like, that's what it comes down to is.

[00:19:56] You know, what's in the data set, what is it that we're consuming? [00:20:00] And quite frankly, it's gross. It is all of the dark parts of the internet. Um, it just newsbroke today, although this has been like circulating for months, um, about the largest image, uh, data set contains child, uh, pornography and child abuse.

[00:20:24] Like that is horrifying, absolutely terrifying that this is what models like mid journey have been trained on. Um, and it, it's a really important thing for people to be aware of in terms of like these bigger conversations around ethics, because part of the problem is the, why, why are we here? Why are we, why is this like happening is because.

[00:20:49] Um, a couple of years ago, the researchers at OpenAI figured out that, Oh, the more data we feed these things, the better they sound, [00:21:00] the more human they sound, the more confident they sound and any like bad stuff that comes out of it. We could just put, um, you know, guardrails in place so that it doesn't spew toxic hate speech and terrible thing.

[00:21:13] But unfortunately, that means that these models essentially magnify. All of the ugly parts of what it is to like live in a society, right? So all of those things live within these massive, uncurated, unconsented data sets. Um, yeah, it's very, very scary. Like it, the hot dog analogy is so perfect in my biased opinion because most people really don't quite understand what goes into something like ChatGPT or MidJourney today.

[00:21:49] I hope that changes over time, but I know that these big tech companies, I mean, that's, they outsource all of this stuff through third parties and do a [00:22:00] really good job kind of covering their tracks and burying some of these stories. But I think it's so important to understand, like, that this stuff is in there because it will continue to sort of leak out, um, in different ways.

[00:22:13] And it already has, right? Like, we've already had tons of problems with some of these things, but. It's so important for, as like, we continue to adopt this technology, the only way we progress with this is with general understanding. The public needs to have some rough fundamental grasp of what these things are even doing to feel comfortable using it or employing it in their work, in their lives.

[00:22:43] Phil Gamache: Yeah. Definitely agree. And I think that you're, you're part two of the LLM guide, like, explains it really well. Like you, you talk about bias first and then the dirty laundry, uh, kind of after. So I, I want to like unpack the, the bias element first. [00:23:00] Um, you talk about, uh, C4, Colossal Clean Crawled Corpus as the massive data set used to train, uh, NLP models like GPT 3 and T5 and how Wikipedia is one of the top contributors in the C4 data set.

[00:23:14] And, you know, Wikipedia is obviously highly biased. The majority of its contributors are 87 percent male, 27 years old, educated, single, without children. This lack of diversity amongst contributors leads to obviously a very narrow viewpoint of a bunch of various subjects. And you highlighted that this, sorry, you highlighted that this leads to several biases like racial, geo, as well as gender, and a few others.

[00:23:45] How can we work towards creating a more balanced and unbiased data set while still leveraging the vast and valuable information available from sites like Wikipedia and maybe like a follow up to that is like how, how can [00:24:00] marketers who are listening to this right now, like ethically used AI tools, aware of the inherent biases to create campaigns that are fair and unbiased, Particularly in like culturally sensitive markets.

[00:24:15] Britney Muller: Yeah. Yeah. Such a good question. And I think the, the way around this would be thoughtfully curated and documented datasets, which we're working on, but we're still like a ways off in terms of having something as vast and as powerful to fuel something like what we see with ChatGPT That, in my opinion, is a step closer because.

[00:24:42] If we even have like rough understanding as to what went into a model, we can know what to look out for, right? We can be more aware of what those issues and problems might be emerging from this technology. Um, and you'd be shocked at how [00:25:00] little documentation there is, even at some of these massive tech companies with the data that they've trained these bigger models on.

[00:25:09] I mean, it's pretty crazy. Yeah. Um, and yeah, you look to something as, like, reputable as Wikipedia, and you see how that's riddled with issues as well. So it's, it really does come down to how can we have a more representative corpus. of what we have in the real world, right? Of mixed views and perspectives and history covering the different people that live in it.

[00:25:36] I mean, Black history alone is like largely, you know, underrepresented on Wikipedia. Um, and as consumers, it's so important to be aware of this. And something that I try to do through my talk, because I feel like It might be most memorable is like through examples and stories of how this has gone wrong previously and what to look out for.

[00:25:59] [00:26:00] Um, one recent thing I came across where it really, um, kind of shocked me that the marketers signed off on this, um, was with the PGA. Yeah, 

[00:26:13] Phil Gamache: I saw that. 

[00:26:14] Britney Muller: Yeah, so the PGA. Released these player headshots that were basically completed, completed the body using AI. And when I first saw this, I was like, how cool is this?

[00:26:28] Like a melding of like two of my favorite things. And then I'm flipping through the images and I get to Tony Finau and they have him and he's a, you know, darker skinned man and they have him basically, it looks like he's standing in a, a dump. It's like all dirt. Dirt mounds in the background. Um, and then the only Asian player is basically seen looking like he's building something, like building something out of like [00:27:00] wood and I don't know, very strange, but they're essentially the only two, you know, non white players that essentially break this.

[00:27:10] Headshot background and the fact that they, you know, I'm assuming just thought it was funny and didn't pause to consider why would this be? And instead let it go through to Instagram. And it also like, Oh my God. Like I think of little kids right now. It's like, how are these things going to shape their worldview?

[00:27:34] Right. So that should have been. Stopped or maybe highlighted as an issue with these models before ever seeing the light of day. So yeah, things like that marketers have to be on high alert for, you know, like, especially it's so clear and eminent with the, with the image generation stuff, um, they're starting to get a little bit better with diversity, but you really have to prime them for it.[00:28:00] 

[00:28:00] Um, but yeah, having that general awareness, I think it's everything. 

[00:28:04] Phil Gamache: Yeah, general awareness is, is one thing. But like you said, I think the, the regulation aspect of it, like before we just like press publish on that stuff, like let's, let's have a team of people who are responsible for. Like double checking stuff, like, Hey, like, how does, how does this look like, how, how might some of the biases in the data set contributed to the output of these images?

[00:28:31] Like, yeah, maybe they're kind of interesting at first glance, but like, Hey, why are all the white golfers have a normal background versus our two colored golfers who like, wait, Oh, let's double check this. It's hit the rerun on mid journey a couple of times, let's get some consistency here. Let's figure some stuff out.

[00:28:52] So yeah, I like the regulation part of like the there's, there's responsibility on, on the marketer's shoulders to when you're [00:29:00] playing around with this stuff, like as, as fun as it might seem and, and addictive also like myself being a mid journey user, like there, there are consequences to, to the stuff that you, you, you put out there from the end results.

[00:29:12] So I think that like, Understanding like the data set and the biases that are in there. I think that like, that's, that's already come like a pretty, uh, like not a super long way, but I feel like chatting with other folks, like they, they understand that part of it, like the bias and the models, like it's, It's easy to understand, like it's, it's easy to forget about it when you're using the tool and you can't really do anything about it, but I feel like there's, there's an either an even darker side of, of LLMs that you talk about in part two, um, I watched the Cleaners, uh, documentary about the people who have to look at the most gruesome and awful content to label that content, uh, like before it goes on, on social sites.

[00:29:58] Uh, so they'd never make it [00:30:00] to those actual platforms. It was absolutely shocking and understandably, you know, no one talks about this stuff, especially on the big tech companies. Open AI used Kenyan workers at less than like two bucks an hour to make Chad GPT less toxic. Like there's so many of these stories that are just like hidden from public site.

[00:30:21] What steps can we take as an industry to like be more ethically minded when we manage and support content moderators who are exposed to like all this harmful material? Like should, should there be an industry wide standard for treatment and support of these content moderators? There's like. How can we get more people to even know about this stuff?

[00:30:43] Like I kind of knew about it, but like when I watched the documentary, I was, I was shocked by how massive the industry was in a specific country that was behind a lot of this work. 

[00:30:53] Britney Muller: It is shocking. It's so disturbing. [00:31:00] And it's. It's crazy as well because it's so well hidden that it, I think it, it is very difficult to kind of surface some of this stuff to regular users who, again, just get the shiny end product, they get the YouTube hot dog, and they have no idea all of, like, the horrendous things that, you know, these cleaners have essentially seen and removed from YouTube.

[00:31:25] Um, it's, it's really tricky. I do hope we get to a point where hopefully AI could help assist with some of this stuff so that we aren't putting humans through this, like, very, uh, traumatic experience. Um, and it's also, yeah, it's just, there's so many interesting layers to it as well, right? Like we have, these are the people that are deciding those edge cases as well.

[00:31:52] Um, and so lines get a little bit blurry and yeah, I don't know. I, I want to [00:32:00] continue, you know, highlighting that so that people are aware and I think we can put pressure on some of the tech companies to, um, be more responsible and how they distribute those tasks and ensure that You know, mental health services are provided that they are, these people are taken care of, or, you know, I don't know, I don't have all the answers for that, but I do think it's a really important area to kind of consider and think about and push back on, quite frankly.

[00:32:37] Phil Gamache: Yeah, so important. Thank you for for making this a part of like all all of your talks and like it's it's it's baked into your guide And yeah, it really opened my eyes to like the the ethics I kind of like knew most of it. But yeah, the the cleaner stuff was was really eye opening and Um, yeah, I, I think the only thing we can do, like you said, like putting [00:33:00] more pressure on the bigger companies to, like, how, how is this a thing, like in, in 2024, like how are we treating people that are doing this type of work like this?

[00:33:10] But yeah, like the, the dirty laundry, the ethics, like there's, there's, there's a dark side to, to, to LLMs and, and, and AI and ML. Um, but there, there are like a lot of like, you know, good things and happier things too. And I think that like, um, one thing that. And my research for this interview stood out. Like I went through a lot of your, your tweets.

[00:33:33] Like I have followed you on, on Twitter for a long time. I think the first time I heard about you was, uh, in Seattle at Moscon. Um, I forget if you were talking about AI or SEO, probably, probably a mix of both, but I've been following you on, on, on Twitter since, and you've got amazing takes, uh, one that you, I think it was a recent tweet, like one of the things that.

[00:33:55] I love the most about marketing operations. Like MarTech is, it's kind of my niche. [00:34:00] It's like this, all this like puzzle solving that's, that's required and, and marketing. And you wrote this compelling tweet that beautifully captures the human essence of problem solving. And the, uh, it was something like the AI, or it's not something like I copy pasted exactly what you wrote.

[00:34:16] So it was the, the idea in the shower, a conclusion in your dream and the solution at dinner. And you have this like fear that AI's dehumanization might potentially rob future generations of this like beautiful, uh, process, if you will. That's, that's very human. I'm only halfway through it, but it resonates a lot with Eric Larson's book, uh, called The Myth of Artificial Intelligence, um, which highlights this idea that like the distinct nature of human intelligence and the overlooked role of abductive reasoning in AI research today.

[00:34:52] And you talked a bit about this at the start of the show with like the the baseball analogy potentially with this here, so I'm curious to learn about that, [00:35:00] but like it's probably just a matter of time until research evolves from deductive and inductive. Inference to embracing abductive inference and enabling this potential form of reasoning, the kind of intuitive problem solving that could potentially be positioned to replace the unique uniquely human experience of like that Eureka moment that you talked about.

[00:35:23] Like, how, how can we maintain this uniquely human aspect of creativity and intuition in the face of like advancing AI tech in the next couple of years? 

[00:35:34] Britney Muller: Yeah, it's a good question. And I, It's so funny because I wrote that tweet because I had such an incredible experience figuring out this very technical problem for a client and I had basically built like a working like prototype to solve it and I knew exactly what I wanted it to be from the get go.

[00:35:55] So I had a clear like end goal, but getting there [00:36:00] took so much hard work, so much trial and error that I was losing my mind. And it, it really was like those moments like at dinner or like I, I had multiple dreams that helped solve like a couple issues I was having. Cause I was just thinking about it nonstop.

[00:36:16] And so I'd had all these like mini moments of like Eureka, like, Oh my God, why did, why haven't I tried this? Like, I need to do this. Um, and I wonder, you know, how many of those we will rob future people of with AI shortcuts. Um, because I, now I could have very easily, you know, have used TrackCPT to get me there a lot faster, but it was a really important and beautiful process for me to, like, figure that out on my own, and it felt so good, um, to do that, um, but, yeah, I mean, it's so funny that you mentioned that book and, you know, Explain a little bit about what it's unpacking in terms of like human thought and that was [00:37:00] something I heard a lot of at NeurIPS last week was like, you know, the thinking fast and slow, the part one and part two of our brain, how can we, you know, better craft these machines to essentially carry some of that?

[00:37:15] And there's some really fascinating information. Thank you. Um, research right now on that, right? Like we know that if we tell a model, large language model to like take time to go through the steps itself, to find the conclusion to a problem, it tends to do better, like asking it to like pause and quote unquote think, right?

[00:37:38] It's not really thinking, but to give it time does lead to better results. And something that. Andrew Ng actually said that I thought was so clever was when you're struggling to get a model to do a particular thing, write out your prompt and give it to a human, right? Like, does it even make [00:38:00] sense to a person?

[00:38:01] And oftentimes we're, we aren't clear enough, right? We aren't, you know, explicitly stating what it is that we're trying to get a model to do. Right. And I think, and this was said really beautifully on, um, your episode with Sarah McNamara, your episode 100. Congratulations, by the way. 

[00:38:21] Phil Gamache: Thank you. 

[00:38:21] Britney Muller: Oh, but I, I love that or she, or you had mentioned how like, uh, stakeholders still aren't clear at asking what you want.

[00:38:32] And so our jobs are totally secure. I'm like, absolutely. Like, that's it. Like that is totally it. And that's what we run into as well. As marketers is like. We aren't always that good at explaining what we must be. And so figuring out how to like really kind of clarify that and streamline it, um, will be good.

[00:38:54] And again, like this technology really, really elevate [00:39:00] when we start connecting it to other functionality. Oftentimes when we're talking about it today in 2023, it's just the large language models. But when connected to different systems. Or models that already do things really well in a deterministic like manner versus probabilistic.

[00:39:19] We can come up with some really, really powerful solutions. Um, but unfortunately, you know, it will be so expensive that I worry that a lot of these capabilities will remain in the hands of the tech giants for now. 

[00:39:31] Phil Gamache: Yeah, maybe not to like the, the level of sophistication that you're talking about, but I feel like one, one area that I go to when you talk about, Combining GPT 4 or Chad GPT with like other sources of data, like.

[00:39:47] OpenAI a couple of months ago released GPTs and the ability to kind of like customize that and pair that with specific niche data sources or, or a training data set that you have access [00:40:00] to that GPT 4 might not have access to, like it, what. What excitement do you have around that stuff? Like we're prepping for an episode on like how we've built our own custom GPTs for the podcast and how to prep for interviews and like repurpose the episode into into a blog post, but how many GPTs has Brittany created so far?

[00:40:23] Are you a fan? 

[00:40:25] Britney Muller: Oh, you're going to hate this answer. I've created exactly zero GPT because that functionality was already available through the API. So you don't need to do any of it. Um, but where I get excited and what I've been playing with is sort of adding onto that functionality in a way that, um, Basically expands capabilities through assistant planning.

[00:40:49] And so, um, you can do some really cool stuff with like the LLM agents where you could have multiple. Um, and that's great for fact checking or reducing quote unquote [00:41:00] hallucination. Um, but sending a specific task like customer support through a really well thought out formula, essentially that has like all the instruction.

[00:41:16] For these LLM assistants to respond really, really well is exciting to me. Um, I saw some really cool examples at NeurEPS. I was like, holy shit. And it's nothing crazy sophisticated, but it's just thoughtful. And I think that is where we see the value add, right? It's not in like these big, sexy, wild things.

[00:41:38] It's going to be in like the time saving opportunities or both Um, you know, help us as marketers, but also help the end user as well. Um, so those, I get really, really excited about those. I also, you know, just in general, I get really excited about different healthcare applications. Um, I [00:42:00] get excited about like wildlife conservation efforts and the stuff that they're doing with drone technology is just so cool right now.

[00:42:07] Um, yeah, there's so many cool applications. The tricky part is like, there's, there's a real lack of resources and financial support for some of what I might deem like the more impactful things that we could do for the world, right? There is total lack of funding for, for things that aren't going to like return all that investment.

[00:42:32] Right. So, uh, something to consider. And I hear this a lot as well on the AI side of things is we're building this technology that has an inverse effect. on, on people with the fewest resources, right? Meaning that those of us with access and power to this technology can essentially, we will wield the benefits first.

[00:42:54] And the people that this technology affects the most through environmental [00:43:00] impact, through some of the biased and ethical issues will, will see the value of it at the very last, like they will be the last to get this technology. And that really, like, Makes me sick. Um, so figuring out ways that we can empower, you know, different populations of people with this technology is super exciting.

[00:43:22] There's, I saw a couple of really cool papers, although not enough on the global South and the different things that AI researchers are doing there to basically empower teachers and students with some of this technology through the use of, um, text. Because they don't have access to, to wifi connection.

[00:43:40] So stuff like that's really, really exciting to me. 

[00:43:43] Phil Gamache: Very cool. Yeah. Thanks for sharing all that. I think that is super powerful. Some very exciting stuff there. Um, one mission that we have on the show is like this idea of feature proofing, uh, marketers and, and, and folks that work in NSEO and marketing ops.

[00:43:59] And I feel [00:44:00] like. A lot of what you're talking about are, are, are things that, you know, might be like passion projects, things that, you know, at some point people might want to like double down on too. And like you, you actually predicted that, uh, on top of the need for basic data science knowledge, like you talked about, like, you know, why aren't we teaching statistical analysis and like basic stag, sig stuff and, and marketing courses.

[00:44:23] You're predicting that the surge in AI may lead to more, uh, potentially Wikipedia spam and intentional, uh, writing errors as human markers, right? So, like, I want to double down on the increased demand for soft skills that you're kind of predicting. How can marketers balance the need to level up their technical chops in data science while still also keeping an eye on upping their people skills?

[00:44:51] Because as you know, the, the, the those will continue to be, uh, important as well. 

[00:44:58] Britney Muller: Yeah, I'm really excited [00:45:00] about those two things converging, though, like the workshop instructor who can lean on otter. ai to record the workshop and, you know, know that we'll have images of whatever the team is working on so that they can be more human, they can be more present.

[00:45:19] Um, same goes for like meetings, I feel like I'm a better communicator with some of this technology that I know is for taking notes for me or for the team. Um, so I get really, really excited about that. And also I think it helps hold people accountable. You know, I don't know if I get confused with Otter and the other one, but one like says who talked the most during the meeting and who asked the best questions.

[00:45:43] Um, that sort of sort of insights is so fascinating and fun for me. Um, Yeah, so I think that that technology is certainly very, very exciting, but I, I continue to challenge marketers that, you know, have traditionally been interested in getting a [00:46:00] bit more technical is like, you really don't have an excuse now, this is the world's best assistant, right?

[00:46:06] Like I used to buy developers lunch all the time, all the time. Or like beers or whatever, just to get like info and like, well, how can I do this? And like, I had, I carried so much guilt from over asking engineers for support and questions that it was like not healthy. And now that I have the freedom to just ask away on a tool that is so good at getting me those insights in the flash of an eye, it's, it's amazing.

[00:46:39] It's, um, I feel like I have a superpower. It's so exciting. And I think people need to really, like, leverage that for what it is that they're working on. And again, like, you don't really have an excuse now because It's just unbelievable what this stuff can do. I mean, you get stuck in Google sheets, trying to do something, go to chat, cheeky T, right?

[00:46:58] Ask it how to do that [00:47:00] task. Um, oh my gosh, I've used it for so many things that it just really, it's incredible. So I think that's exciting to get more people like technically savvy and building things that they wouldn't otherwise be able to do. And then on the flip side, I think the soft skills will be more important than ever, ever.

[00:47:21] And let me tell you what, we need soft skills in the AI industry. There is a large population of people building this technology with very high IQs and very low EQs. I don't mean that to diminish any of these people in general. Like I'm just, it's, that's an observation. And quite frankly, I, I would put myself in the boat in, in terms of, it's not going to be any of those people that.

[00:47:48] I've been using this technology up until this point to think of the brilliant next steps. I'm, I get super, super excited about kind of being a catalyst for some of this information [00:48:00] so that domain experts like you can come up with like the brilliant Martech solution that I would have never thought about or never considered.

[00:48:09] And it's even like, you know, like the stay at home moms and, you know, um, people caring for a loved one who's going through a medical. Call. Like there's, there's real world applications that I want to see. solutions for, right? Like how can we just in general make people's lives easier, better, more efficient.

[00:48:30] Um, and again, I think empowering people, non technical people to make some of those like application ideas or considerations is what's going to level all this up because it traditionally, and unfortunately all the attention goes to kind of the VPU provider, right? It's all in the AWS and Google Cloud, but like.

[00:48:54] In order for those to even stay afloat and available, we need applications to, like, blow up and [00:49:00] be successful and provide real world value, um, while obviously, like, mitigating harm and being very careful about how we do it. But yeah, so I, I get really, really excited about that stuff and even, you know, learning about different applications Becomes more and more evident how essential the domain experts are.

[00:49:19] That's the bottleneck in AI. It's no longer data and no longer AI researchers. It's domain experts, not crazy. I think that's so exciting. Um, so I hope people can kind of like get excited about that and really. Kind of learn the fundamentals to have that power of like, Going about your day and realizing, Oh, like this could definitely be automated, right?

[00:49:45] We could use something like what I just learned about to do this. This 

[00:49:50] Phil Gamache: is a fantastic answer. I think there's like a blueprint for future proofing your job, but just like upscaling and like rethinking the way [00:50:00] that you're, you're working right now. And it's not just like learning how to double down on, on the people skills, but also like the, the access that you have to expand your.

[00:50:09] Your technical skills with, with tools like ChatGPT. Brittany, this has been a fascinating conversation. I feel like we, we've done a good job balancing the excitement also, like the, we need to think about the darker side of things also. Um, if you've listened to a couple episodes, you know, that we end all of our, uh, interviews with one question.

[00:50:30] You're an outdoor adventurer, an avid snowboarder, an international keynote speaker, a writer, an SEO consultant. You've got a lot going on in your life. One question we ask all our guests is how do you remain happy and successful in your career? How do you find balance between all the things you're working on while staying happy?

[00:50:47] Britney Muller: That's, I love this question. I've been thinking about this for days now and I don't even know how to answer that. And the more I thought about it, the more I realized that like. I, [00:51:00] I don't prioritize happiness, I really work hard to prioritize meaning. And so for me, that means quality over quantity, uh, and it means like really loving and feeling passionately about the work that I'm doing and that it will have a positive effect on others.

[00:51:19] That, to me, at the end of the day is so much richer and more fulfilling than, you know, filling my days with dopamine hits. Um, so I, I mean, it's obviously important, you know, prioritize like fun and happiness, but I really like, I, I feel like there was a period in my life where I went quite a while without feeling like I was doing meaningful work and meaningful things and being able to identify that and be like, Ooh, that's, you know, that's not a healthy thing.

[00:51:53] And for me personally, I really sort of need that in my life. So that, I would say that, [00:52:00] and that feel. Sometimes better than like just pure happiness in my statement is like, Oh shit. Like we were able to build this thing that really helped all these people do this. Or that makes me really, really excited.

[00:52:11] And so I try to use that as like the North star. 

[00:52:16] Phil Gamache: Very cool. Super powerful. Brittany, it's been super fun. Uh, anything else you want to plug for folks? I'll share links to, uh, your guide, uh, data science, uh, one on one, uh, anything else you want to plug? 

[00:52:29] Britney Muller: Not really. I think, you know. Books are cheat code, like, people have like forgotten about books, I feel like, and there's so many great reads out there that can help support people through, like, these changing times.

[00:52:44] Um, and so, yeah, some of those, I guess, like, resisting AI is really, really good. Um, There's a bunch I can send you links to and we can throw them in the 

[00:52:57] Phil Gamache: Sweet. Awesome. Yeah, let's do that. [00:53:00] Brandy, thank you so much for your time. This was super fun, very powerful conversation. Uh, yeah, super important stuff, but also exciting, crazy times to be living right now.

[00:53:10] Britney Muller: Yeah, for sure. Thank you so much, Phil.