Pod Insights Voices of Search // A Search Engine Optimization (SEO) & Content Marketing Podcast LLM.txt files — trend or trash? About this episode Google's AI overviews now appear in over 50% of search results. Duane Forrester, founder and CEO of UnboundAnswers.com and former Microsoft search engine insider, brings two decades of industry perspective to navigating this transformation. The discussion covers essential skill development for AI-era SEO including structured data mastery for LLM consumption, chunking content strategies that balance machine readability with human engagement, and critical evaluation frameworks for emerging AI SEO tools that prioritize trustworthiness over feature quantity. See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.
Transcript 0:00 Speaker 00 The Voices of Search Podcast is a proud member of the I Hear Everything Podcast Network. Looking to launch or scale your podcast? I Hear Everything delivers podcast production, growth, and monetization solutions that transform your words into profit. Ready to give your brand a voice? Then visit ihereeverything.com. Welcome to the Voices of Search Podcast, a member of the I Hear Everything Podcast Network. Ready to expedite your company's organic growth efforts? Sit back, relax, and get ready for your daily dose of search engine optimization wisdom. Here's today's host of the Voices of Search Podcast, Tyson Stockton.
0:43 Speaker 01 One more, and this one's, this will be a short form one. So this one, again, we're going to look at Trend or Trash, L M T X T files, Trend or Trash.
0:53 Speaker 02 Ooh, that sounds fun. Trash. Total trash. They did not get their answer out fast enough. Total trash, right? Let's be crystal clear. Every crawler out there follows the robot T X T protocol. That's the protocol. All crawlers follow that protocol. Robots dot T X T. L M dot T X T is an invention, kind of like domain authority from Moz. It's their own makeup. The engines don't look at domain authority to make any decisions. They don't factor that into anything on their end. In fact, you could say that, um, oh, what was the, Google had this for years. Um, the name for their D A was rank. It's escaping me now.
1:37 Speaker 02 I haven't talked about it for so long. Holy cow. It was one to 10. They had a rank rating for you and everybody was chasing it. We had it in a little toolbar at the bottom and it was like, you know, this was every, it was like, if you had a higher seven out of 10, then I wanted a link from you back to me and that would help boost my score and all of that, right? Um, oh, page rank. And, um, those types of things are useful in that they are visualizations of a complex topic. Okay. For years, I wanted to. I wanted to create one when I was at Bing. Um, the problem was the scale up Bing was, I think it was like one to sixty two thousand or something.
2:18 Speaker 02 So it didn't make sense. You, you had to translate that into something that was like rational for the human mind to wrap their head around. You know, because like, how do I tell you you score four hundred and sixty two thousand? What does that even look like on a graph? You barely see the color. Like, it just looks heartbreaking, you know? Um, but there was never any value in exposing it because what the page rank score taught us was that people wanted to gain scores like that. They use them for tracking their gamification of like, oh, well, if I get a link from here, then that'll boost mine. If I get a bunch of those links, that boosts mine, that boosts my ranking. And even though I don't deserve to be ranking well, I am ranking well.
2:59 Speaker 02 And I'll take the traffic and the revenue. And as long as I don't pop on a radar in Mountain View, I'll be okay. Like, and I think everybody's learned from those bad old days and kind of moved on. Um, but the fact of the matter is, um, things like the L L N dot T X T or those are, those are made up. Um, conceptually, I get it. Somebody looked at it and went, oh, robot T X T is for the old crawlers. We need one for the new crawlers. No, you don't, because the new crawlers are still just crawlers. And they follow robot T X T. And point in fact, C C bot, the Creative Commons bot, is actually responsible for a whole bunch of the data that makes its way into these systems. Because all of the systems access the data from Creative Commons bot.
3:43 Speaker 02 So if it's following robot T X T, you don't need anything else. That's it. Maybe in the future, but right now I'm gonna toss that in the trash.
3:53 Speaker 01 Yeah, it's like you, you can create whatever T X T file you want. Doesn't mean anyone's gonna look at it. You can create them for sure.
4:00 Speaker 02 Totally, totally. And you know what? On that point, I think this is really important as well. So for your site maps and your robot T X T, spend the time to get to learn how the actual syntax should look, what, what you should be putting into those files, especially robot T X T. You know, I'll skip the old warning about how easy it is to block everything on your website and like screw yourself because, you know, you put slash in there in the wrong spot, right? But understand what control in there looks like. The default for every robot is do crawl. That is the default for a bot. It's its only reason for existence is to crawl a URL. So there is no do crawl directive.
4:44 Speaker 02 There is a do not call directive where you tell it what not to go into. So go through and understand all of that properly because I see robot txt files daily with all of this, go here, go here, go here. The crawlers ignore all that crap. You're building it, maintaining it. You think you're having some kind of influence on it? Nothing. If you didn't have that, it wouldn't impact the crawler at all. It's gonna go. If it's a crawlable URL and it can find it, it's gonna go. That's its only reason for existence. So like, understand how those things matter because when they do make updates or they do make changes to those things, it's really important to understand and the impact they have. So there you go. And everybody audit your robot txt.
5:29 Speaker 02 If somebody in your IT department is blocking the bots from these AI systems, oh, buy them a pizza, have a conversation. You need to get that role back. They need access.
5:40 Speaker 01 Absolutely. So that's going to wrap up this episode of the Voice of Search Podcast. Thanks again to Duane Forrester from Unbound Answers for joining us. If you'd like to get in contact with Dwayne, you can find a link to his LinkedIn profile in their show notes, or you can get more information about Unbound Answers at unboundanswers.com. Also be sure to check out his Substack, which will be in the show notes for some of these articles that have been referenced. And if you haven't subscribed yet and would like a daily stream of SEO content marketing knowledge in your podcast feed, hit the subscribe button on your podcast app or on YouTube and we'll be back in your feed soon. With that, that's all for today. Thanks for stopping by and we'll see you in the next episode.
Previous Does retrieval-based search make traditional keyword research obsolete? Next Invest SEO resources building comprehensive content vs acquiring high-quality backlinks Page updated November 15, 2026 · transcript and summary are machine-generated