What is robots.txt
Robots.txt is a plain text file at the root of a website.
It tells crawlers which URLs they may request. Search engines and AI bots fetch it before they crawl anything else, and the file uses it to ask them to skip certain paths. It can't force a bot to obey, and it doesn't remove pages from search results. Robots.txt contains crawl requests that well-behaved bots choose to follow.
Robots.txt defined for SEO
In SEO, robots.txt is the file that decides which parts of a site search engine bots spend their time on. It follows the Robots Exclusion Protocol. Martijn Koster proposed the protocol in 1994, and the Internet Engineering Task Force (IETF) later published it as RFC 9309. Technical SEO teams use the file to keep bots away from low-value URLs, such as internal search results or filtered category pages. Make sure it sits at yourdomain.com/robots.txt, because crawlers don't look for it anywhere else. Each subdomain needs its own file.
What the file actually does
Robots.txt tells crawlers which URLs they shouldn't request, and that's all it does. Google Search Central says the file mainly stops crawlers from overloading your server and "is not a mechanism for keeping a web page out of Google." Links from other sites allow a disallowed page to appear in results as a bare URL with no description. To keep a page out of search, use a noindex tag or a password. Don't put a noindex tag on a URL you've also blocked, because the crawler never fetches the page and never sees the tag.
How crawlers read it
A crawler looks for the group of rules addressed to its own name and follows only that group. Each group opens with a User-agent line, followed by Disallow and Allow lines that list URL paths. If no group names the bot, it falls back to the group marked with an asterisk. When two rules match the same URL, Google applies the more specific one, which means the rule with the longer path. Paths are case-sensitive, so /Blog/ and /blog/ are different folders.
Blocked by robots.txt explained
"Blocked by robots.txt" means a crawler wanted a URL, but your file told it not to fetch that page. Google Search Console shows this status in its page indexing report. Sometimes the block is deliberate, as with admin pages or cart URLs. A leftover "Disallow: /" from a staging site, or a rule that blocks your CSS folder, can hide pages you want ranked. Audit each flagged URL against the file and confirm you meant to block it.
Can robots.txt be ignored
Yes. Any crawler can ignore robots.txt, because the file has no way to enforce its rules. Google notes that Googlebot and other respectable crawlers obey it, but "other crawlers might not." RFC 9309 adds a second warning: listing a path in the file makes it public and discoverable. So never list a private folder in robots.txt to hide it. Protect that folder with proxy or server authentication.
Robots.txt for AI crawlers
AI crawlers read robots.txt the same way search bots do, and each AI company publishes the user-agent names you can address. A company that runs several bots with different jobs needs separate rules to cover them all.
Training crawlers
OpenAI's GPTBot collects content for model training. OpenAI says that blocking GPTBot indicates your content shouldn't be used for that training. Google handles AI training opt-outs through a separate token, Google-Extended. Blocking that token doesn't touch Googlebot's search crawl.
Search and assistant crawlers
OAI-SearchBot fetches pages for ChatGPT search, and OpenAI treats each bot setting as independent. You can allow OAI-SearchBot while you block GPTBot. Before you block any AI bot, decide whether you want your pages to appear in AI assistant answers. Blocking a search crawler can keep you out of them.
A minimal robots.txt example
In this example, the first group lets every crawler in, except to the admin area and internal search pages. The second group keeps GPTBot off the whole site. An empty "Disallow:" line allows crawling, while "Disallow: /" blocks everything. One slash separates those two outcomes, so test every change before you publish it.
User-agent: *
Disallow: /admin/
Disallow: /search
User-agent: GPTBot
Disallow: /
Sitemap: https://www.example.com/sitemap.xmlRelated SEO terms
Robots.txt controls crawling.
Knowing what is robots.txt, and what it isn't, keeps you from using it for a job another tool does better. Open yourdomain.com/robots.txt now and read every Disallow line. If a line blocks a page you want found, delete that line.
Look up more AI search and SEO terms in our glossary.