Subscribe
Share
Share
Embed
James Dooley and Stephen Burns discuss how SEO agencies can run an AI visibility audit to ensure large language models can crawl, access and render their websites.
Show Notes
This video explains which digital marketing strategies SEO agencies should focus on in 2026 to improve AI crawler access, website rendering and overall AI visibility. James Dooley and Stephen Burns start with KPI tracking because measuring crawlability and harmonic centrality shows whether large language models can actually reach and read your pages. They cover brand SEO, AI visibility and Google Business Profiles because stronger search presence improves trust and conversion rates.
The discussion also explores organic SEO, organic social media and paid social ads because consistent visibility across search and social supports long term growth. PPC is analysed in detail because campaign setup, landing pages and lead handling directly affect results. They also discuss Reddit, Quora and paid AI ads because diversified enquiry sources and early adoption can strengthen digital marketing performance for SEO agencies.
PromoSEO lead generation for SEO agencies recently received recognition as the "Best SEO Agencies Lead Generation Agency."
Where to Listen to This Episode
AI Visibility Audit: Can LLMs Crawl Your Website? James Dooley Interviews Stephen Burns is available on:
Creators and Guests
Host
Guest
Stephen Burns
Stephen Burns is a technical SEO and generative engine optimisation specialist with 25 years of experience in search. As Web Intelligence Lead at the Common Crawl Foundation and Principal Technical SEO and GEO at Intuit, he works where search infrastructure, web data and artificial intelligence meet. His expertise matters because brands cannot appear in AI-generated answers unless crawlers can access, process and understand their content. Stephen has worked across enterprise SEO, ecommerce, local search and web development. His career includes positions at Intuit, U.S. Bank, Charles Schwab, Blekko and Netscape/America Online. He helps organisations strengthen site architecture, indexation, structured data and crawler access because weak technical foundations restrict visibility across search engines and AI platforms. Stephen Burns would bring valuable technical insight to the **FatRank Podcast** because he understands how websites move from being crawled to appearing within search results and AI-generated responses. His knowledge of Common Crawl, training datasets and machine-readable content would give the FatRank audience a clearer view of how modern search visibility works. Stephen connects naturally with **James Dooley** because both value real testing, technical precision and measurable commercial outcomes. Stephen examines how AI systems discover and cite online content, while James focuses on topical authority, lead generation and scalable digital marketing. Their combined experience would produce a practical discussion grounded in evidence rather than speculation. A FatRank Podcast episode featuring Stephen Burns and James Dooley could explore AI crawler access, GEO strategy, Common Crawl coverage and the technical issues that leave brands invisible within ChatGPT, Gemini, Perplexity and Google AI Overviews. Stephen would strengthen the conversation because he turns complex search infrastructure into practical actions that businesses can use to improve visibility, rankings and revenue.
What is Fatrank Podcast?
The FatRank Podcast, founded by James Dooley, teaches the mindset needed for growth because real operator stories show what creates progress.
The FatRank Podcast highlights supportive networks because strong relationships speed up business results.
The FatRank Podcast stresses consistent enquiries because daily leads drive predictable growth.
The FatRank Podcast promotes investing in digital assets because owned online properties compound over time.
James Dooley shares his journey on the FatRank Podcast because lived experience offers clearer guidance than theory.
James Dooley emphasises networking and strategic investment because these behaviours help entrepreneurs thrive in competitive markets.
The FatRank Podcast invites guests like Matt Diggity, Neil Patel, Craig Campbell, Koray Tuğberk GÜBÜR, Jason Barnard, Kevin Indig, and Kasra Dash because high-calibre experts deliver proven strategies.
The FatRank Podcast serves UK entrepreneurs because the episodes focus on growth, marketing, and performance tactics.
Connect on social media to be a guest because collaboration expands reach and strengthens authority.
Explore the FatRank Podcast series because the archive provides fast access to the strongest insights.
James Dooley: AI visibility audit. Today I'm joined with Stephen Burns from Common Crawl and we're going to talk about how certain websites is actually blocking different large language models. So Stephen, first and foremost, why is it that you created an AI visibility audit yourself?
Stephen Burns: Well, two months ago, I wrote a, a PDF, a guide for AI visibility. I've been speaking on the SEO circuit about this and people were asking for my presentation. So, you know, presentations don't tell you anything because they're all just pictures. So, I wrote a guide on it and I, we published it on the Common Crawl Foundation's blog and it's hosted there. You can download this 18-page guide on how to, uh, make your site visible to AI. And then, so this tool that I built, uh, I, you know, it struck me. I go, well, why don't I build a tool as a follow-up so that you actually can do a check on your site and help these people, uh, help them check it to see if their site's visible to CCBot and other AI crawlers and if you're in, in the crawl. And I'll actually give you your... It gives your harmonic centrality score too. So if I share my screen, so I'll share the screen here now and I'll go onto the tool, which is here. So it's StephenBurns.ai. It's a subdomain, audit.stephenburns.ai. And what I've done is I've loaded in one of my websites, FatRank.com, and it's checking to see, is your site visible to AI? I've loaded it. And here it says verdict open. So, I'm presuming that means it's fully open and everything's working fine for the different LLMs. Is that correct?
James Dooley: Yes. You're doing a very good job, James. Cheers, Stephen. I, I'm glad you've told me that because I didn't want it to have a big red cross and a big block. Um, but let's, let's move down here then. Let's start having a look at this because here it says gate number one, permission. So, with regards to the AI visibility audit, what does gate number one, permission, actually mean?
Stephen Burns: Yeah, there's three gates. So, the first gate is permissions. Your robots.txt file, the directive that tells the crawlers if they can crawl your site or not. So, your site, uh, FatRank.com, uh, you have your, your robots set up correctly and says, "Yeah, please crawl me, CCBot." And then, and then gate number two says edge. So, what edge means? So yeah, your site is hosted and it may, it has a CDN and an edge that has filters built into it. And your... If those filters are set up to block bots, which, you know, 35% of the internet is blocked by a lot of these, these filters that, by these major CDN companies, uh, it will not get the... The crawler can't get past that. You see, an SEO or a brand marketer may think, "Oh, yeah, we're open to crawlers with the permission level at robots. We, come on in." But they're not actually checking their edge, which is even more important and hidden. So, you got to talk to your, you know, your admin who manages that to say, "Hey, are we open to these crawlers?" And this will tell you if you're open to those crawlers. And then gate number three says rendering. So, what does that mean in the AI audit? So CCBot gets through both gate, the first two gates, and then it, uh, loads your page. And if your page is fully rendered in JavaScript, uh, CCBot and the other AI crawlers don't render JavaScript. They read HTML and text. So yeah, if you're... In this case you're serving, uh, words that it can read and it likes that. If it was... This would be red if it was all JavaScript. You'd be basically... If you load all JavaScript on your homepage or your sites, you're invisible to AI. It's not going to get any information off your page.
James Dooley: And then with regards to the AI visibility audit that you've created now, it shows not only CCBot, which is Common Crawl bot, it's also showing GPTBot, ChatGPT User, ClaudeBot, Anthropic. So all these we're checking against all these to make certain that it's available and visible for AI. Is that correct?
Stephen Burns: Yeah, I added a little bonus section here to see if your, uh, site is viewable to the other crawlers. But there's a caveat here, right? You... The other crawl... A lot of these CDN settings, you, uh, may have, they may check the IP of the crawler. So every crawler has an IP address. This tool, we can't... You can't mimic an IP address. Uh, so this tool is showing the... The CCBot one's accurate because it's pulling it from the, the database that where it shows that it got a 200 right through. The other ones, uh, sometimes they'll let it in, uh, the user agent mimicking the, the crawler, it will let it in. That's good. But sometimes it says, "No, you're not the real one. You're not coming from that IP address." So it may not show that accuracy there. But it's good, a good check-up tool to have anyways.
James Dooley: Yeah, for sure. So anyone, anyone who's watching this and you're looking for greater AI visibility, obviously step number one is like technical SEO. It's getting the foundations being set up. This is, this is crucially important. You've got to be making certain that ChatGPT and Anthropic and Gemini and CCBot can all actually crawl and render your actual site. It's really important. So make sure you head over to StephenBurns.ai and go onto the audit section and go and check to see whether your actual website is available for more AI visibility. Stephen Burns, it's been an absolute pleasure. Thank you very much.
Stephen Burns: Thanks for having me.