Overview
A robots.txt file controls which parts of a website search engine crawlers are allowed to access, while a meta robots tag controls whether an individual page should be indexed and how it can appear in search results. Use robots.txt to manage crawling and use a meta robots directive such as noindex when you want a page to remain accessible to crawlers but stay out of search results.

 

The purpose of a robots.txt file, also known as the robots exclusion protocol, is to tell search engine crawlers which areas of a website they are allowed or not allowed to crawl. A meta robots tag works differently, giving search engines page-level instructions about whether a page should be indexed and how its content should appear in search results.

Understanding that difference is important because crawling and indexing are two separate processes.

A robots.txt file usually sits at the root of your website, for example:

 

www.domain.com/robots.txt

 

Before a compliant search engine crawler requests a URL on your website, it can check this file to see whether crawling that area is permitted.

A simple robots.txt rule might look like this:

 

User-agent: *

Disallow: /

The User-agent: * tells compliant crawlers that the rule applies to all robots.

The Disallow: / then prevents those crawlers from requesting any pages on the website.

 

That is a particularly important example because one incorrect character in a robots.txt file can potentially stop search engines from crawling an entire website.

Robots.txt should therefore be treated as an important technical SEO file rather than something that is set once and forgotten.

 

What is the difference between robots.txt and meta robots?

This is where a lot of confusion arises.

Robots.txt and meta robots can both influence how search engines interact with your website, but they control different things. The most important distinction is this:

Use robots.txt when you want to control crawling.

Use a meta robots noindex rule when you want to control whether an accessible webpage appears in search results.

Google specifically states that a robots.txt file is not the appropriate method for keeping a normal webpage out of Google Search. If you want to stop a webpage being indexed, noindex is generally the appropriate instruction.

There is an important catch.

Google needs to crawl the webpage in order to see its meta robots tag.

If you place:

 

<meta name=”robots” content=”noindex”>

 

on a page and then block that same page through robots.txt, Google may be unable to crawl the page and therefore unable to see the noindex instruction.

That distinction is one of the most important things to understand when deciding between robots.txt and meta robots.

 

What does a good robots.txt file look like?

There is no single “best robots.txt” file that works for every website.

The appropriate setup depends on the website, its CMS and which sections search engines need to crawl.

A fairly straightforward example might look like this:

 

User-agent: *

Disallow: /wp-admin/

Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.example.com/sitemap.xml

 

In this example:

  • User-agent: * applies the rules to all compliant crawlers
  • /wp-admin/ is blocked from crawling
  • the admin-ajax.php file is specifically allowed
  • the XML sitemap location is declared separately.

 

A larger ecommerce website may have additional rules controlling URLs generated by filters, search functions or other areas that create large numbers of low-value URLs.

The important point is that rules should be added for a specific reason.

A longer robots.txt file is not automatically a better one.

 

Why should a website have a robots.txt file?

A robots.txt file gives website owners a way to manage how compliant crawlers access different areas of the website.

Even if you are happy for search engines to crawl almost everything, the file can still be useful for declaring the location of your XML sitemap and providing clear instructions around areas that do not need to be crawled.

Robots.txt can become particularly useful on larger websites.

For example, an ecommerce site may generate thousands of URLs from:

  • filters
  • sorting options
  • internal search results
  • faceted navigation
  • session parameters
  • duplicated URL combinations.

Allowing crawlers to spend large amounts of time accessing pages with little search value can make the site unnecessarily complicated to crawl.

Robots.txt can sometimes help control this.

However, blocking a URL from crawling should always be done carefully. If a page needs to rank, search engines generally need to be able to access its content.

 

Can robots.txt stop a page appearing in Google?

Not necessarily.

This is one of the most common misunderstandings around robots.txt.

A robots.txt rule can stop Googlebot from crawling a URL, but it does not necessarily stop Google from knowing that the URL exists.

Google may discover the URL from internal links, external links, sitemaps or other sources.

In some situations, the URL could therefore still appear in search results without Google having crawled the content itself.

If your objective is specifically to prevent a webpage from being indexed, a noindex directive is generally the more appropriate approach.

 

What are the limitations of robots.txt?

Robots.txt is useful, but it has several limitations that website owners should understand…

 

Can robots.txt protect confidential information?

No.

Robots.txt should never be used as a security measure.

The file is publicly accessible, meaning anybody can usually visit:

 

www.example.com/robots.txt

 

and see which directories have been listed.

If you have private or confidential information that should only be available to authorised users, use proper access controls such as password protection or authentication.

Blocking the URL from search crawlers does not make the content private.

 

Do all crawlers obey robots.txt?

No.

Reputable search engines generally respect recognised robots.txt rules.

However, malicious or poorly behaved crawlers can simply ignore them.

This is another reason robots.txt should not be relied upon for security.

 

Can different crawlers interpret instructions differently?

Potentially.

Search engines broadly support the robots exclusion standard, but some crawler-specific behaviour can vary.

If a rule is being created for a particular crawler, check the documentation for that crawler before relying on unusual or advanced directives.

For important SEO decisions, keeping the file simple can often reduce the likelihood of unintended consequences.

 

What robots.txt directives can you use?

There are several common ways robots.txt rules can be configured.

 

How do you allow all robots to crawl a website?

A simple file could contain:

 

User-agent: *

Disallow:

 

An empty Disallow line means there is no crawling restriction being applied by that rule.

 

How do you block all compliant robots from a website?

You would use:

 

User-agent: *

Disallow: /

 

This blocks crawling across the domain and should be used with extreme care on a live public website.

 

How do you block a particular folder?

For example:

 

User-agent: *

Disallow: /private-folder/

 

This tells compliant crawlers not to crawl URLs within that folder.

 

How do you block a particular crawler?

A crawler-specific rule could look like:

 

User-agent: ExampleBot

Disallow: /

 

This applies the restriction to that crawler rather than every bot.

 

How do you declare an XML sitemap?

You can include:

 

Sitemap: https://www.example.com/sitemap.xml

 

The sitemap location should be written as the full absolute URL.

 

Are robots.txt directives case-sensitive?

URL paths used in robots.txt rules are case-sensitive.

For example:

 

Disallow: /images/logo.gif

 

may not necessarily apply in the same way to:

 

/Images/Logo.gif

 

if the website treats those as separate paths.

That means the wording of URL paths should reflect the actual URLs used on the website.

The robots.txt filename itself should also be available at the root of the website as:

 

/robots.txt

 

Can you use wildcards in robots.txt?

Major search engines including Google support pattern matching within robots.txt rules.

The * character can be used to represent a sequence of characters.

For example:

 

Disallow: /*?sort=

 

could be used in an appropriate setup to restrict crawling of URLs containing a particular sorting parameter.

The $ character can indicate the end of a URL.

For example:

 

Disallow: /*.gif$

 

would target URLs ending in .gif.

Wildcards can affect many URLs at once, which makes them powerful but also means mistakes can have a large impact.

Before adding a broad rule, check examples of affected URLs and confirm that valuable pages will remain crawlable.

 

How can you check whether robots.txt is blocking a page?

Google Search Console can help diagnose crawling problems.

The URL Inspection tool can show whether Google is able to access a particular URL and can flag situations where crawling is prevented by robots.txt.

If a page that should be ranking is not being crawled, checking robots.txt should be one of the technical checks carried out.

This is also an area that should be reviewed during a wider SEO audit.

A technical audit should look beyond whether a robots.txt file exists. It should check whether the rules match what the business actually wants search engines to crawl.

 

What is a meta robots tag?

A meta robots tag is an HTML instruction that tells search engine crawlers how a particular page should be treated once they access it.

A typical noindex tag looks like this:

<meta name=”robots” content=”noindex”>

It is usually placed within the <head> of the webpage.

This tells search engines that support the directive not to include that page in their search results.

Google also supports several other robots directives that control indexing and how content can be displayed.

 

What are the main meta robots directives?

Some of the most commonly used directives include:

 

Index

index allows a page to be indexed.

This is the default behaviour, so explicitly adding index is not usually necessary.

 

Noindex

noindex tells supporting search engines that the page should not appear in their search results.

For example:

<meta name=”robots” content=”noindex”>

 

Nofollow

nofollow tells the crawler not to follow links found on that page.

For example:

<meta name=”robots” content=”nofollow”>

This is different from adding a rel=”nofollow” attribute to an individual link.

 

Noimageindex

noimageindex tells Google not to index images from that page.

 

Nosnippet

nosnippet prevents Google from displaying a text or video snippet for the page in search results.

This is worth understanding in the context of modern search because Google states that nosnippet also prevents the affected content from being used as a direct input for AI Overviews and AI Mode.

That makes robots directives relevant beyond traditional blue-link search results.

 

Can you use more than one meta robots directive?

Yes.

Multiple instructions can be combined within the same meta robots tag.

For example:

 

<meta name=”robots” content=”noindex, nofollow”>

 

This tells supporting search engines not to index the page and not to follow links from it.

However, these directives should not be added automatically.

For many pages where noindex is required, allowing search engines to follow the links on the page may still be appropriate.

The correct setup depends on the purpose of the page.

 

Should you use robots.txt or noindex to remove a page from Google?

If the objective is to prevent an HTML webpage from appearing in Google Search, noindex is usually the appropriate instruction.

The page needs to remain crawlable so that Google can access and read the tag.

A common technical mistake is to do both of the following:

 

Disallow: /example-page/

 

and:

 

<meta name=”robots” content=”noindex”>

 

This can create a conflict in the intended setup.

If Googlebot obeys the robots.txt rule, it cannot crawl /example-page/. If it cannot crawl the page, it cannot see the noindex meta tag.

For that reason, do not block a page in robots.txt simply because you have added a noindex tag and want it removed from search.

Google needs access to the page to process the indexing instruction.

 

When should you use robots.txt instead of meta robots?

Robots.txt is generally more appropriate when the goal is to control crawler access to a section or pattern of URLs.

Examples could include:

 

  • internal search result URLs
  • unwanted parameter combinations
  • particular administrative areas
  • certain duplicated or low-value crawl paths
  • crawler-specific restrictions

A meta robots tag is generally more appropriate when you are happy for a search engine to access the page but do not want that page appearing in search results.

Examples might include:

  • certain confirmation pages
  • particular internal utility pages
  • pages that need to remain accessible to users but should not appear in organic search

The decision should always be based on the outcome you are trying to achieve.

 

Is robots.txt the same as a robot tag?

No.

The wording is similar, which is partly why people confuse them.

robots.txt is a text file at the root of the website that primarily controls crawler access.

A robots meta tag is an instruction placed on an individual HTML page that can control indexing and search-result behaviour.

There is also an X-Robots-Tag, which provides similar indexing controls through an HTTP response header.

X-Robots-Tag can be particularly useful for non-HTML resources such as PDFs, where inserting an HTML meta tag is not possible.

 

What is the best robots.txt setup for SEO?

The best robots.txt file is usually one that does only what the website actually needs.

For a relatively small website, that may mean a very short file.

For a large ecommerce or publishing website, more detailed crawling controls may be justified.

The aim is not to block as much as possible.

The aim is to allow search engines to crawl valuable pages while avoiding unnecessary crawling where there is a genuine technical reason to do so.

When reviewing a robots.txt file, ask:

  • Are any important pages accidentally blocked?
  • Are search engines able to crawl pages we want indexed?
  • Are there large groups of unnecessary URLs being crawled?
  • Is the sitemap correctly declared?
  • Are old rules still present from previous site versions?
  • Have development or staging rules accidentally reached the live site?
  • Are we trying to use robots.txt to solve an indexing problem that actually needs noindex?

These checks are particularly important after website migrations, redesigns and changes to CMS configuration.

 

Why should robots.txt be checked as part of an SEO audit?

Robots.txt is only a small file, but its reach can be enormous.

One incorrect rule can affect a directory containing hundreds or thousands of URLs.

This means it should be reviewed during technical SEO audits, particularly where a business has experienced unexplained drops in visibility or where important pages are missing from Google’s index.

A broader SEO audit can also identify related problems involving:

  • Indexing
  • canonical tags
  • redirects
  • XML sitemaps
  • internal linking
  • duplicate URLs
  • crawlability
  • response codes

Robots.txt rarely exists in isolation from the rest of a site’s technical setup.

Understanding how it interacts with meta robots directives is therefore much more useful than simply knowing how to write a Disallow rule.

 

What should you remember about robots.txt and meta robots?

Robots.txt primarily manages crawling. Meta robots primarily manages indexing and search-result behaviour.

If you want to stop compliant crawlers requesting a section of your website, robots.txt may be appropriate.

If you want Google to crawl a webpage but prevent that page appearing in search results, a noindex meta robots directive is usually the better choice.

And if the content is genuinely private, neither method should be treated as security. Use proper access controls instead.

A small technical mistake in either area can affect how search engines access and understand a website, so these instructions should always be checked carefully before they are implemented.