OpenAI says robots.txt may not apply to one of its bots
A reminder to check which OpenAI bot you are blocking, because only one of them controls whether you appear in ChatGPT.
What happened?
OpenAI documentation for its web crawlers states that ChatGPT-User, the agent that fetches a page when someone asks ChatGPT a question, is not used for crawling the web in an automatic fashion, and that because those actions are initiated by a user, robots.txt rules may not apply. Separately, TollBit reported in its State of the Bots review of the first half of 2026 that roughly 15 percent of AI page fetchers in Europe reached URLs that site owners had marked as disallowed, and that ChatGPT-User reached disallowed pages on more sites than any other bot it tracked.
Why does it matter?
Two things follow. First, a robots.txt disallow is a request rather than a lock, and vendors read it differently: Anthropic states that its bots honor industry standard directives in robots.txt, while OpenAI carves out user-initiated fetches. Second, and more useful to you, the bot that decides whether you show up in ChatGPT search answers is OAI-SearchBot, not ChatGPT-User. OpenAI documentation says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Blocking the wrong bot can cost you visibility without buying you much control.
Should I do anything?
Open the plain text file at yourdomain.com/robots.txt and read it once, looking for a single thing: whether OAI-SearchBot is disallowed. If it is, and you want to be findable in ChatGPT, that is your one change to make this week. Leave every other rule alone for now. If you have no robots.txt file at all, nothing is being blocked and there is nothing to fix. If you cannot find or edit the file, ask whoever manages your website to check that one line for you.
Learn more
A note on legal topics: this update is general information, not legal advice. If your organization needs to act on a regulation, ruling, or policy change, check the official source linked above and consult a qualified attorney.
More updates
View allChatGPT is aiming more of its searches at named domains
ChatGPT now scopes many searches to specific sites, which means the shortlist forms before the search runs.
Read updateSEOs report PDFs dropping out of Google results
Practitioners are seeing PDF clicks fall sharply in Search Console, with HTML pages showing instead.
Read updateStudy: AI engines cite trade media far more than mainstream news
A study of 129 million AI citations reports that niche and trade outlets get cited well ahead of mainstream press.
Read update