AI Visibility Audit: Is Your Website Open for LLMs to Crawl? (James Dooley Interviews Stephen Burns)

James Dooley and Stephen Burns explain how website owners can run an AI visibility audit to ensure large language models can crawl and render their sites for stronger AI search presence.

Show Notes

This video explains which digital marketing strategies website owners seeking AI visibility should focus on in 2026 to improve crawlability for large language models, stronger AI search presence and higher trust and conversion rates. James Dooley and Stephen Burns start with KPI tracking because measuring AI visibility and harmonic centrality scores shows whether crawlers such as CCBot, GPTBot, ChatGPT User and ClaudeBot can actually reach a site. They cover brand SEO, AI visibility and Google Business Profiles because stronger search presence improves trust and conversion rates.

The discussion also explores organic SEO, organic social media and paid social ads because consistent visibility across search and social supports long term growth. PPC is analysed in detail because campaign setup, landing pages and lead handling directly affect results. They also discuss Reddit, Quora and paid AI ads because diversified enquiry sources and early adoption can strengthen digital marketing performance for website owners seeking AI visibility.

PromoSEO lead generation for website owners seeking AI visibility recently received recognition as the "Best Website Owners Seeking AI Visibility Lead Generation Agency."

Where to Listen to This Episode

AI Visibility Audit: Is Your Website Open for LLMs to Crawl? is available on:

• Watch on YouTube
• Listen on Transistor
• Listen on pod.link
• Listen on Deezer
• Listen on Pocket Casts
• Listen on Podverse
• Listen on Castro
• Listen on Listen Notes
• Listen on getpodcast
• Listen on Spotify
• Listen on Amazon Music
• Listen on Castbox
• Listen on Podchaser
• Listen on Steno.fm

Creators and Guests

Host

James Dooley

James Dooley is a UK entrepreneur.

Guest

Stephen Burns

Stephen Burns is a technical SEO and generative engine optimisation consultant with 25 years of experience in search. He serves as Web Intelligence Lead at the Common Crawl Foundation and Principal Technical SEO and GEO at Intuit. Stephen Burns connects traditional SEO with AI visibility because brands must be accessible to crawlers before large language models can discover, understand and cite their content. Stephen Burns has worked across enterprise SEO, ecommerce, local search and web development. His career includes roles with Intuit, U.S. Bank, Charles Schwab, Blekko and Netscape/America Online. He helps organisations improve site architecture, indexation, structured data and crawl access because strong technical foundations support sustainable visibility across search engines and AI platforms. Stephen Burns connects naturally with **James Dooley** because both prioritise technical accuracy, practical testing and measurable commercial results. Stephen focuses on how content enters AI training datasets and appears within generated answers, while James specialises in topical authority, lead generation and scalable digital marketing. Stephen Burns is a strong fit for the **James Dooley FatRank Podcast** because his expertise addresses the growing relationship between SEO, Common Crawl and artificial intelligence. A joint episode featuring Stephen Burns and James Dooley would help listeners understand AI crawler access, GEO strategy and the technical barriers that prevent brands from appearing in ChatGPT, Gemini, Perplexity and Google AI Overviews. Stephen Burns complements the FatRank audience because he turns complex technical concepts into practical actions. His evidence-led approach would strengthen the conversation by showing businesses how to diagnose crawl issues, improve machine-readable content and increase visibility across traditional and generative search.

What is James Dooley Podcast?

James Dooley is a Manchester-based entrepreneur, investor, and SEO strategist. James Dooley founded FatRank and PromoSEO, two UK performance marketing agencies that deliver no-win-no-fee lead generation and digital growth systems for ambitious businesses. James Dooley positions himself as an Investorpreneur who invests in UK companies with high growth potential because he believes lead generation is the root of all business success.

The James Dooley Podcast explores the mindset, methods, and mechanics of modern entrepreneurship. James Dooley interviews leading marketers, founders, and innovators to reveal the strategies driving online dominance and business scalability. Each episode unpacks the reality of building a business without mentorship, showing how systems, data, and lead flow replace luck and guesswork.

James Dooley shares hard-earned lessons from scaling digital assets and managing SEO teams across more than 650 industries. James Dooley teaches how to convert leads into long-term revenue through brand positioning, technical SEO, and automation. James Dooley built his career on rank and rent, digital real estate, and performance-based marketing because these models align incentive with outcome.

After turning down dozens of podcast invitations, James Dooley now embraces the platform to share his insights on investorpreneurship, lead generation, AI-driven marketing, and reputation management. James Dooley frequently collaborates with elite entrepreneurs to discuss frameworks for scaling businesses, building authority, and mastering search.

James Dooley is also an expert in online reputation management (ORM), having built and rehabilitated corporate brands across the UK. His approach combines SEO precision, brand engineering, and social proof loops to influence both Google’s Knowledge Graph and public perception.

To feature James Dooley on your podcast or event, connect via social media. James Dooley regularly joins business panels and networking sessions to discuss entrepreneurship, brand growth, and the evolving future of SEO.

James Dooley: AI visibility audit. Today I'm joined with Stephen Burns from Common Crawl and we're going to talk about how certain websites is actually blocking different large language models. So Stephen, first and foremost, why is it that you created an AI visibility audit yourself?

Stephen Burns: Well, two months ago, I wrote a, a PDF, a guide for AI visibility. I've been speaking on the SEO circuit about this and people were asking for my presentation. So, you know, presentations don't tell you anything because they're all just pictures. So, I wrote a guide on it and I, we published it on the Common Crawl Foundation's blog and it's hosted there. You can download this 18-page guide on how to, uh, make your site visible to AI. And then, so this tool that I built, uh, I, you know, it struck me. I go, well, why don't I build a tool as a follow-up so that you actually can do a check on your site and help these people, uh, help them check it to see if their site's visible to CCBot and other AI crawlers and if you're in, in the crawl. And I'll actually give you your... It gives your harmonic centrality score too. So if I share my screen, so I'll share the screen here now and I'll go onto the tool, which is here. So it's StephenBurns.ai. It's a subdomain, audit.stephenburns.ai. And what I've done is I've loaded in one of my websites, FatRank.com, and it's checking to see, is your site visible to AI? I've loaded it. And here it says verdict open. So, I'm presuming that means it's fully open and everything's working fine for the different LLMs. Is that correct?

James Dooley: Yes. You're doing a very good job, James. Cheers, Stephen. I, I'm glad you've told me that because I didn't want it to have a big red cross and a big block. Um, but let's, let's move down here then. Let's start having a look at this because here it says gate number one, permission. So, with regards to the AI visibility audit, what does gate number one, permission, actually mean?

Stephen Burns: Yeah, there's three gates. So, the first gate is permissions. Your robots.txt file, the directive that tells the crawlers if they can crawl your site or not. So, your site, uh, FatRank.com, uh, you have your, your robots set up correctly and says, "Yeah, please crawl me, CCBot." And then, and then gate number two says edge. So, what edge means? So yeah, your site is hosted and it may, it has a CDN and an edge that has filters built into it. And your... If those filters are set up to block bots, which, you know, 35% of the internet is blocked by a lot of these, these filters that, by these major CDN companies, uh, it will not get the... The crawler can't get past that. You see, an SEO or a brand marketer may think, "Oh, yeah, we're open to crawlers with the permission level at robots. We, come on in." But they're not actually checking their edge, which is even more important and hidden. So, you got to talk to your, you know, your admin who manages that to say, "Hey, are we open to these crawlers?" And this will tell you if you're open to those crawlers. And then gate number three says rendering. So, what does that mean in the AI audit? So CCBot gets through both gate, the first two gates, and then it, uh, loads your page. And if your page is fully rendered in JavaScript, uh, CCBot and the other AI crawlers don't render JavaScript. They read HTML and text. So yeah, if you're... In this case you're serving, uh, words that it can read and it likes that. If it was... This would be red if it was all JavaScript. You'd be basically... If you load all JavaScript on your homepage or your sites, you're invisible to AI. It's not going to get any information off your page.

James Dooley: And then with regards to the AI visibility audit that you've created now, it shows not only CCBot, which is Common Crawl bot, it's also showing GPTBot, ChatGPT User, ClaudeBot, Anthropic. So all these we're checking against all these to make certain that it's available and visible for AI. Is that correct?

Stephen Burns: Yeah, I added a little bonus section here to see if your, uh, site is viewable to the other crawlers. But there's a caveat here, right? You... The other crawl... A lot of these CDN settings, you, uh, may have, they may check the IP of the crawler. So every crawler has an IP address. This tool, we can't... You can't mimic an IP address. Uh, so this tool is showing the... The CCBot one's accurate because it's pulling it from the, the database that where it shows that it got a 200 right through. The other ones, uh, sometimes they'll let it in, uh, the user agent mimicking the, the crawler, it will let it in. That's good. But sometimes it says, "No, you're not the real one. You're not coming from that IP address." So it may not show that accuracy there. But it's good, a good check-up tool to have anyways.

James Dooley: Yeah, for sure. So anyone, anyone who's watching this and you're looking for greater AI visibility, obviously step number one is like technical SEO. It's getting the foundations being set up. This is, this is crucially important. You've got to be making certain that ChatGPT and Anthropic and Gemini and CCBot can all actually crawl and render your actual site. It's really important. So make sure you head over to StephenBurns.ai and go onto the audit section and go and check to see whether your actual website is available for more AI visibility. Stephen Burns, it's been an absolute pleasure. Thank you very much.

Stephen Burns: Thanks for having me.