Most businesses assume that ranking well in Google means they exist everywhere that matters. Good position on page one, solid domain authority, years of backlinks. Case closed.
Except a growing share of research now happens somewhere Google can’t fully see: inside a chat window. A page can rank beautifully in traditional search and still be functionally invisible to the systems answering questions there.
We wanted to know where our own clients actually stood on AI search visibility before Q4 hit. So we ran an internal audit to check whether AI search could actually get in the door.
Why We Started Asking “Can AI Search Even Read Our Sites?”
Lumping “SEO” and “AI search” together feels natural. Same alphabet soup, same invisible algorithms, same general anxiety about being found. The similarities stop there.
Traditional search hands someone a list of links and lets them choose. AI search picks a handful of sources, decides what to say, and speaks on a brand’s behalf.
Same league, different game.
For B2B companies, the stakes aren’t theoretical. A recent study tracking ChatGPT-referred traffic across nearly a thousand e-commerce sites found that for complex, considered purchases, referrals from ChatGPT outperformed several traditional marketing channels. Marketing services qualify as about as “complex purchase” as it gets.
ChatGPT alone accounts for more than nine in ten AI-referred visits across the study, which makes it the one system worth getting right first.
If a prospect is already asking ChatGPT to shortlist vendors before ever finding a contact form, that’s the exact shift we mapped out when we wrote about GEO becoming the new frontier for AI search. This audit was our way of finding out whether our clients were even part of that conversation.
What We Actually Checked
The list was more plumbing than glamour: robots.txt permissions for known AI crawlers, XML sitemap accuracy, Core Web Vitals scores, and whether pages were returning clean responses instead of quietly failing.
We also checked something most audits skip: crawlability vs. indexability. A page can be crawlable but excluded from an index, or indexable in theory but unreachable in practice. Both failure modes look identical from the outside.
Schema markup got checked too, since structured data works like a translation layer, telling machines what a page is actually about instead of leaving them to guess. A few pages had none at all, despite ranking respectably in traditional search.
Here’s the part that trips people up. OpenAI’s own documentation states that its crawler must be allowed to access a page before it can even read a noindex tag on that page. Block the crawler entirely, and the “don’t index this” instruction never gets delivered. The block and the message can’t both travel through the same locked door.
What We Found
One client’s robots.txt file was blocking an AI crawler that had been renamed since the file was last updated two years ago.
A second client’s XML sitemap still pointed to product pages retired eighteen months earlier, quietly feeding outdated URLs to any system trying to map the site.
Neither business had a reason to think about these files again once they were set. That assumption turned out to be the actual problem, showing up in some form across nearly every site in the audit.
The pattern held even wider than our own client list. Independent research on scraper behavior found that some categories of bots, AI search crawlers included, frequently ignore robots.txt instructions altogether, regardless of what a site actually requests. Getting robots.txt right for AI search, in other words, is not a one-time task.
It’s something that quietly expires.
How The it Crowd Fixes AI Crawlability Before It Becomes a Q4 Problem
So what does actually fixing this look like? Not a redesign. Not a rebuild.
A structured pass through the exact plumbing most agencies never check: crawler permissions verified against current user-agent lists, sitemap accuracy confirmed, Core Web Vitals brought back into range, and indexing directives tested against real crawler behavior instead of assumed compliance.
That’s AI search readiness in practice: unglamorous maintenance work that decides whether a business shows up when it actually matters.
The same discipline applies to the infrastructure work we’ve done helping Dallas businesses future-proof their websites heading into a new year, nothing glamorous, everything load-bearing.
Being technically accessible doesn’t guarantee a business gets chosen. It just means the business is allowed in the room.
Imagine a store with the best inventory in town, gorgeous windows, a five-star reputation, and a locked front door with no sign, no buzzer, and no bat-signal. Customers assume it’s closed. They’re not wrong, exactly. They just can’t tell from the sidewalk.
That’s what an unreachable robots.txt file does to an otherwise excellent website. Nobody’s questioning the product. They just can’t get past the door.
Fixing it doesn’t take a miracle, or even much drama. Mostly, it takes someone willing to go check the lock.
Go check yours before Q4 finds out first.
