What Is a robots.txt File, and What Should Yours Say?

Short answer: a robots.txt file is a small text file at the root of your website that tells search engine crawlers which pages or folders they shouldn't crawl. It doesn't hide pages from search results or protect private content. Most small business sites need only a short file that allows crawling and points to the sitemap.

What does a robots.txt file actually do?

Search engines use automated programs, often called crawlers, to visit websites and read their pages. Before a well-behaved crawler starts, it looks for a file called robots.txt and reads any instructions in it.

Those instructions are simple: which crawlers they apply to, and which parts of the site those crawlers should not visit. Think of it as a polite sign on certain doors saying "please don't go in here". Reputable search engines respect it. It is a request, though, not a lock.

Where does it live, and how do you check yours?

The file always sits at the root of your domain, so you can see it by typing your address followed by /robots.txt into a browser, for example yourdomain.com/robots.txt.

If you see a short list of lines, that's your file. If you get a "page not found" message, your site doesn't have one, which simply means crawlers are free to visit everything.

What should a small business robots.txt include?

For most small business websites, the file should be short and permissive:

• Allow crawling by default. You want search engines to read your service pages, product pages, blog posts and contact page.
• A sitemap line. One line giving the full address of your XML sitemap helps crawlers find all your important pages.
• Block only pages with no search value. Typical examples are admin or login areas, shopping cart and checkout pages, and internal search results pages.

A simple file might look like this:

User-agent: *
Disallow: /admin/
Disallow: /cart
Sitemap: https://www.yourdomain.com/sitemap.xml

The first line means "these rules apply to all crawlers". Each Disallow line names a path to skip, and the last line points to the sitemap.

What can robots.txt NOT do?

This is where most confusion comes from:

• It doesn't remove pages from search results. If a blocked page is linked from elsewhere, search engines may still list its address, just without a description, because they weren't allowed to read it.
• It isn't a noindex instruction. To keep a page out of results, the page needs to be crawlable and carry a noindex tag, so search engines can read that instruction.
• It isn't security. The file is public. Anyone can read it, and listing private folders in it can actually point people towards them. Private content needs a login or password.

Which robots.txt mistakes can remove a site from search?

A single wrong line can have a big effect:

• A leftover "Disallow: /" from a staging build. Developers often block a whole test site while it's being built. If that line goes live with the real site, it tells crawlers to skip every page.
• Blocking style and image files. Search engines need to load a page's layout and images to understand it. Blocking those folders can make pages look broken to them.
• Blocking pages you want found. A rule meant for one folder can accidentally match more than intended, for example blocking a whole section of products or posts.

If your pages have suddenly stopped appearing in search, robots.txt is one of the first things to check. Our guide on why search engines may not be indexing your pages covers the other common causes.

How does robots.txt fit with your sitemap and canonical tags?

These pieces work together, and they should agree:

• Your sitemap lists the pages you want found. Nothing in the sitemap should be blocked in robots.txt. See what an XML sitemap is and how to submit it.
• Your canonical tags tell search engines which version of a page is the main one. A canonical should never point to a page that robots.txt blocks. See what a canonical tag is.
• robots.txt keeps crawlers away from areas with no search value, so their time is spent on the pages that matter.

How do you test a change before it goes live?

Because the file affects your whole site, test every change:

1. Write the new rules and read them back slowly, checking each path.
2. Use the robots.txt testing or inspection features in your search engine's webmaster tools to check that important pages are still allowed.
3. After publishing, open yourdomain.com/robots.txt to confirm the live file is the one you meant.
4. Over the following weeks, watch your page indexing reports for any unexpected drop.

Not sure what your robots.txt is telling search engines?

We check and set up robots.txt, sitemaps and canonical tags as part of our technical basics. See our technical search set-up service, or talk to us on WhatsApp.