The answer first: blocking OpenAI's crawler GPTBot in robots.txt only stops OpenAI from using your website content to train its models. It does not make your website disappear from ChatGPT's search answers. Search is handled by a different crawler, OAI-SearchBot. OpenAI's official documentation is direct about this: the two settings are independent of each other, and a webmaster can allow OAI-SearchBot in order to appear in search results while blocking GPTBot to indicate that the content should not be used for training.
What really makes a website disappear from ChatGPT is blocking the wrong one, or blocking all three together. This article separates the three crawlers one by one, then gives robots.txt rules for four options. The full list of AI crawlers, how to trace accidental blocking by a CDN and how to verify with logs are already covered in the tool page article "AI Crawlers Explained", so they are not repeated here.
What Each of the Three OpenAI Crawlers Does
OpenAI accesses websites with three different user-agents, each with its own purpose. Everything below is taken from OpenAI's crawler documentation page.
GPTBot: training. OpenAI states that it is used to crawl content that may be used to train its generative AI foundation models, and blocking it indicates that the site's content should not be used for training. It is the best known of the three, and also the one most often blocked by mistake. Be aware that the name "GPTBot" collides in search results with a company that has a similar name, so this article writes "OpenAI's crawler GPTBot" when it first mentions it.
User-agent example: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
OAI-SearchBot: search inclusion. OpenAI states that it is used to make websites appear in the results of ChatGPT's search features, and it says explicitly that sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers and can only appear as navigational links. OpenAI recommends allowing it in robots.txt and allowing the officially published IP ranges.
User-agent example: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
ChatGPT-User: a user opening a link in real time. When a user asks a question in ChatGPT or a custom GPT, it may visit a web page in real time with this user-agent. OpenAI states that it does not crawl a whole website automatically. Because the action is started by a user, robots.txt rules may not apply. It also does not decide whether content can appear in search, and search is managed through OAI-SearchBot.
User-agent example: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
In one sentence: GPTBot governs what future models know, OAI-SearchBot governs whether you appear in answers today, and ChatGPT-User is the program sent to read the page when a user opens your link.
What Happens When You Block the Wrong One
Treating the three as one thing usually ends in one of two ways.
The first is intending to block training but blocking search. A webmaster hears that "AI will take content for training" and blocks all three in one go. The result is that when a customer asks ChatGPT "which company in Hong Kong provides web hosting", this website does not appear in the answer, and its competitors do. The training part is indeed blocked, but the cost is losing exposure in ChatGPT answers straight away.
The second is believing search is blocked when it is not. Some websites block only ChatGPT-User and assume ChatGPT will then not cite them. According to OpenAI's documentation, ChatGPT-User does not decide search inclusion, and user-initiated actions may not be bound by robots.txt, so writing the rules this way blocks nothing at all.
There is also a time lag to know about. OpenAI states that after a robots.txt change, the search systems take about 24 hours to adjust. Seeing no change on the day you edit the file is normal.
robots.txt Rules for Four Options
First, one robots.txt rule that causes many cases of self-blocking. According to Google's robots.txt specification, each crawler applies only one group of rules, namely the group whose user-agent matches it most closely. A crawler-specific group and the * general group are not merged. In other words, if your robots.txt has User-agent: * with Disallow: / and has no separate group for OAI-SearchBot, OAI-SearchBot will apply the general group and the whole site is blocked. Conversely, once a crawler has its own group, it looks only at that group.
The four blocks below can be copied directly.
Option 1: fully open. Training and search are both allowed. This suits company websites and content websites that want to be cited by AI as much as possible.
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /Option 2: block training only. This keeps exposure in ChatGPT search but keeps content out of training. For most company websites, it is the safer choice.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /Option 3: block everything. Training, search and user-initiated visits are all refused. This suits only websites that are sure they do not want to appear in ChatGPT, such as internal systems or member-only areas. Bear in mind that ChatGPT-User is triggered by users and OpenAI's documentation states that robots.txt may not apply, so this group only states a position.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /Option 4: search only. Only OAI-SearchBot is allowed and the other two are refused. The only difference from Option 2 is ChatGPT-User, and the practical effect is close to Option 2.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Disallow: /Whichever you choose, confirm that these groups are written outside the User-agent: * group, and that robots.txt itself can be opened directly in a browser. Once you have decided to allow access, you can also add an llms.txt in the root directory to help models grasp the site's key points faster. For how to write it, see the tool page article "llms.txt Guide". For whether to add one and how to check it, see "What Is llms.txt?" in this series.
Other Crawlers Follow the Same Logic
Separating training and search into two crawlers is common practice across model providers. Only the crawlers described in official documentation are listed below. For the full list, see the tool page article "AI Crawlers Explained".
| Company | Training use | Search or user use | Key points from official documentation |
|---|---|---|---|
| Anthropic | ClaudeBot | Claude-SearchBot, Claude-User | All three respect robots.txt. Blocking ClaudeBot means future content is not included in training |
| Perplexity | None | PerplexityBot, Perplexity-User | PerplexityBot is for search only and is not used to train foundation models. Perplexity-User is triggered by users and generally ignores robots.txt |
| Google-Extended | Googlebot | Google-Extended is only a robots.txt control and has no separate crawler. Blocking it does not affect inclusion or ranking in Google Search | |
| Apple | Applebot-Extended | Applebot | Pages that block Applebot-Extended can still appear in search results |
The Google row deserves one more sentence. Google Search Central states that content control for AI Overviews and AI Mode works through Googlebot's robots.txt and controls such as nosnippet. Google-Extended governs whether content may be used to train Gemini models, and whether it may be used as grounding when the Gemini apps and Vertex AI generate answers. Blocking Google-Extended does not make a website disappear from AI Overviews. Controlling AI Overviews requires a different set of methods.
A Point for Hong Kong Websites: Your CDN May Already Have Blocked Them for You
However open your robots.txt is, if requests are already stopped at the CDN or firewall layer, the crawler never reaches your server. Cloudflare announced on 1 July 2025 that new domains are asked during setup whether to allow AI crawlers, and that the default changed from allowing AI crawlers to leaving the decision to the site owner. Cloudflare also said that since it launched one-click blocking in September 2024, more than one million customers have chosen to block AI crawlers. If your website sits behind a service such as Cloudflare, you may have shut OAI-SearchBot out without knowing it, and you should confirm this once in the provider's control panel.
How to Check Whether You Have Blocked the Wrong One
Three methods, from simple to thorough.
- Read robots.txt. Open
https://your-domain/robots.txtand look for the three groupsGPTBot,OAI-SearchBotandChatGPT-User, and whether theUser-agent: *group hasDisallow: /. - Check the server logs. Search for the three user-agent strings above and look at the status codes: 200 is normal, 403 means blocked. To verify that the source is real, compare against the IP lists published by OpenAI (gptbot.json, searchbot.json, chatgpt-user.json). If there is a CDN in front of the website, requests blocked by the CDN do not appear in the origin server logs and must be checked in the CDN's control panel.
- Scan once with a free tool. AEO Auditor checks robots.txt, AI crawler access and structured data together. Enter a URL and the result appears in about 30 seconds.
Blocking and allowing are both legitimate choices. The only question is whether you have blocked the wrong target. Once the three crawlers are clearly separated, write robots.txt according to your own position, and you will avoid the situation of "thinking only training was blocked, when customers were blocked too". Allowing crawlers is only the first step. For how to write content that AI will cite afterwards, see the earlier "GEO guide" on this site.
Check for free now whether your website blocks the wrong AI crawlers
AEO here means Answer Engine Optimisation. It has nothing to do with the Authorized Economic Operator Programme of the Customs and Excise Department.
Frequently Asked Questions
Does Blocking GPTBot Make a Website Disappear From ChatGPT?
No. GPTBot collects training content, and whether a website can appear in ChatGPT's search answers is decided by OAI-SearchBot. OpenAI states that the two settings are independent of each other.
How Soon Does ChatGPT Search Adjust After a robots.txt Change?
OpenAI states that after a robots.txt change, the search systems take about 24 hours to adjust.
Can Blocking Only ChatGPT-User Stop ChatGPT From Citing a Website?
No. According to OpenAI's documentation, ChatGPT-User does not decide search inclusion, and user-initiated actions may not be bound by robots.txt. ChatGPT search is managed through OAI-SearchBot.
For more questions, see the tool's FAQ page.
Sources
Accessed: 2026-10-17
- OpenAI, "Overview of OpenAI Crawlers" (the purposes of the three crawlers, user-agent strings, IP lists, the 24-hour adjustment period, and how ChatGPT-User relates to robots.txt): https://developers.openai.com/api/docs/bots
- Google Search Central, "How Google interprets the robots.txt specification" (how user-agent rule groups are applied): https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec
- Google Search Central, "Overview of Google crawlers and fetchers" (definition and scope of Google-Extended): https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
- Google Search Central, "AI features and your website" (how content is controlled for AI Overviews and AI Mode): https://developers.google.com/search/docs/fundamentals/ai-optimization-guide
- Anthropic Help Centre, "Does Anthropic crawl data from the web, and how can site owners block the crawler?": https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- Perplexity, "Perplexity Crawlers": https://docs.perplexity.ai/docs/resources/perplexity-crawlers
- Apple Support, "About Applebot": https://support.apple.com/en-us/119829
- Cloudflare press release, "Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large" (2025-07-01): https://www.cloudflare.com/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/