Fourteen days of production access logs and OAI-SearchBot never requested a product URL. Googlebot hit the same templates four hundred times. ChatGPT still answered the category query from a 2021 roundup on a domain we do not control.
That is the gap chatgpt search visibility actually lives in. Not a missing FAQ schema. Not another title-tag pass. The search bot never fetched the live HTML, so the model had nothing of ours to quote.
If you are the technical marketer trying to appear in ChatGPT search, start with the crawler, the page it can parse, and the proof in the logs. Everything else is decoration.
Allow the search bot, not only GPTBot
Training crawl and search crawl are not the same job. GPTBot is the training crawler. ChatGPT-User shows up when a person asks the product to visit a URL. OAI-SearchBot is the one that retrieves pages for ChatGPT search.
A robots.txt that only names GPTBot leaves the search path blocked or undefined. A blanket `Disallow: /` for unknown bots does the same. WAF rules that treat “GPT” as a scraper’s signature often drop OAI-SearchBot before Apache or Nginx ever logs the request.
The current user-agent strings, and which bot is for training versus retrieval, sit in the OpenAI bot documentation. We copy those strings into robots.txt and into the WAF allowlist. We do not copy them from a screenshot in Slack.
Allow OAI-SearchBot on the URLs you actually want cited: category templates, product detail, the guides that hold unique numbers, process, and named constraints. Do not allow it only on the blog and then wonder why the commercial pages never appear as source pages.
Keep the sitemap clean enough that a retrieval crawl can find those URLs without walking three layers of faceted junk.
What OAI-SearchBot actually downloads
OAI-SearchBot does not run your React hydration. It does not wait for the cookie banner. It does not execute the GTM container that injects the H1.
It issues a GET. It takes the bytes in the first response. If those bytes are a shell, a spinner, or a “enable JavaScript” stub, that is the page. ChatGPT search visibility cannot be better than that payload.
We curl the public URL with the search bot’s user-agent, then we curl it as a normal browser. If the two HTML documents do not share the same H1, the same first paragraph, and the same facts, the bot is reading a different site from the one you edit.
CDN HTML caches keyed only on a desktop Chrome cookie will serve the bot a stale or empty variant. Edge workers that rewrite body copy for logged-in users will do worse. The fetch that matters is anonymous, bot-identified, and impatient.
Rendered HTML versus the editor preview
The block editor lies. The staging preview lies. The Lighthouse lab test on a warmed cache lies in a different way.
Rendered HTML, for this job, means the document a cold GET returns with OAI-SearchBot in the User-Agent header. Headings, main copy, tables, and the canonical tag have to be in that document. Client-side routers that paint the article after a JSON round-trip do not count.
When the front end is a client-rendered app, no amount of chatgpt seo copy in the CMS reaches the bot. The fix is development that returns the article in the first HTML response, not another round of meta descriptions on an empty `#root`.
Server-side render, static generation, or a traditional template that prints the body on the server: any of those can work. A marketing page that only exists after three API calls will not.
Check pagination, filters, and “load more” patterns the same way. If the second page of the category is only a fetch in the browser, OAI-SearchBot never sees those products.
Source pages ChatGPT can quote without guessing
Retrieval wants a page that states a fact in plain language, with enough surrounding context that the sentence can stand on its own. Thin category copy that repeats the H1 in three synonyms does not give the model anything to cite.
Source pages that survive this are specific. A number. A constraint. A sequence. A named part. A condition under which the advice does not apply. That is the texture ChatGPT search can point at.
Duplicate the same 90-word blurb across twenty locations and the bot has no reason to prefer your URL over a scraper that stole it in 2023. Unique does not mean long. Unique means the claim is not already sitting on three other hosts.
We pick a small set of URLs per site: the pages that should be the answer, not the pages that are easiest to publish. Then we make sure those URLs return 200, canonicalise to themselves, and do not share a title with a tag archive.
Do not hide the useful paragraph behind an accordion that never makes it into the initial HTML. If the proof is in a PDF, publish the same facts as HTML. Retrieval still prefers a page it can parse without a plugin.
Proof in the logs, not a chat screenshot
A colleague pasting a ChatGPT answer into Slack is not proof of chatgpt search visibility. It is a single session, on a single prompt, with a retrieval set you did not control.
Proof is boring. Access logs that show OAI-SearchBot fetching the right paths, with 200s, on a cadence that matches how often those URLs change. A stored copy of the HTML it would have received. A note of which URL, if any, ChatGPT named when we ran a fixed prompt set from a clean account.
Grep for `OAI-SearchBot`. Confirm the status codes. Confirm the URLs are the canonical ones, not the session-id variants, not the AMP leftovers, not `?utm_`. If the bot only hits `/` and `/blog/`, the commercial templates are still invisible.
Search Console will not save you here. It does not report this crawler. Treating a green Core Web Vitals report as evidence of ChatGPT retrieval is how teams waste a quarter.
When a citation does appear, archive the answer, the quoted fragment, and the live HTML of that source page on the same day. If the live HTML no longer contains the fragment, you do not have a visibility win. You have a stale index of a page you already changed.
CDN rules, WAF and silent blocks
The failure we see most often is not robots.txt. It is a security layer that never writes a 403 the app can see.
Cloudflare Bot Fight Mode, a “block empty user agents” rule, a rate limit on data-centre ASNs, a country allowlist that assumes buyers only, a managed challenge on anyone who is not Chrome: any of those can drop OAI-SearchBot with no line in the origin log.
If origin logs are empty for that user-agent, read the CDN logs. If the CDN logs are empty, the request never reached you. Allowlist the published bot, then re-test with a fetch that uses the real UA.
Hosting in the Philippines behind a WAF tuned for brute-force login noise is a common stack on the sites we inherit. The same ruleset that stops `wp-login.php` floods will treat a retrieval crawl as abusive if it is keyed on request rate and ASN, not on a signed bot identity.
Do not “allow all OpenAI” as a wildcard and then forget it at the next firewall clone. Pin the user-agent, pin the purpose, and re-check after every CDN change.
Canonicals, parameters and the URL that gets cited
ChatGPT will quote the URL it retrieved. If that URL is a parameterised sort order, a print view, or a duplicate with a trailing slash mismatch, that is the URL a user sees in the citation.
Self-canonical every page you want as a source. 301 the junk. Do not leave `www` and naked host both returning 200 with identical bodies. Do not leave HTTP and HTTPS both live.
Pagination needs a consistent story: either the paginated URLs are worthy source pages, or they canonicalise to a view that actually contains the content. Infinite scroll that never exposes a crawlable page two is a retrieval dead end.
Hreflang does not replace a clear canonical. If we ship a British English guide and a Philippines page on the same topic, each needs its own substance. A cloned template with the city name swapped is not two source pages. It is one weak page plus a duplicate.
WordPress output that survives a bot fetch
If the live stack is WordPress, the HTML OAI-SearchBot receives is whatever the theme, the cache plugin, and the page builder emit on a cold GET. Visual editors that store the real copy in a JSON blob and print a placeholder in `the_content()` fail this test.
That is why our WordPress development work starts with a curl of the public URL, not with the block editor. We disable the layer that serves bots a different skin from humans, then we confirm the H1 and the first two paragraphs are in the raw response.
Object cache plus a full-page cache plus a delayed JS bundle is a normal WordPress pile. It is also how a “works in Chrome” page becomes an empty document for a retrieval bot. Purge paths after publish. Do not rely on a human hitting the URL to warm the only cache variant that contains the article.
Watch SEO plugins that rewrite canonicals on tag archives, that noindex the pages you meant to rank in ChatGPT search, or that inject the same meta description on every product. The plugin is not the strategy. The bytes in the response are.
Migrations make this worse. A leftover redirect map that sends the new guide to a 2020 blog post will train retrieval on the old URL. We crawl the destination host as the bot, not as ourselves, before we call a cutover done.
The checklist we run before we call it visible
We do not declare chatgpt search visibility from a single lucky citation. We run a short, ugly list.
robots.txt names OAI-SearchBot and allows the money paths. The WAF and CDN allow that UA without a challenge. Origin and edge logs show fetches on those paths with 200s.
A cold GET with the search bot UA returns the same core copy a human sees. Rendered HTML includes the H1, the facts, and a self-referencing canonical. No login wall, no cookie interstitial in the first response, no “enable JavaScript” body.
Five to fifteen source pages exist that could actually answer the queries we care about. They are not duplicates. They are not PDFs. They are not three sentences above a form.
A fixed prompt set, run from a clean session, is archived with dates. When a URL of ours is cited, the quoted fragment still exists on the live page. When it is not cited, we do not invent a story about “the algorithm.” We look at whether the bot ever fetched the page.
If any row on that list fails, more content will not fix it. Publishing through the failure just gives Googlebot more HTML that OAI-SearchBot still never sees.
Decide the next step, then scope the audit
Open the last fourteen days of logs and search for OAI-SearchBot. If it is missing, fix allowlisting before you touch copy. If it is fetching `/` and ignoring the templates that hold the answers, fix internal linking, sitemaps, and robots. If it is fetching the right URLs and the HTML is a shell, the stack is the work.
Pick the small set of source pages that should be cited. Make each one a page a retrieval crawl can quote without guessing. Then re-fetch them as the bot and keep the HTML.
When the blocker is the front end, the cache, or a WordPress output path that will not print the article on a cold GET, a scoped audit is the next useful thing. Bring the logs, the robots file, and the URLs you need as source pages. We will tell you whether the next move is a ruleset change, a render fix, or a rebuild of the templates the bot actually hits.
