A website can have excellent articles, useful tools, and well-designed pages, but search engines still need to discover and crawl its content before those pages can become eligible to appear in search results.
One small file helps website owners manage this process: robots.txt.
A robots.txt file is a text file placed in a website’s root directory that gives instructions to participating search engine crawlers about which URLs they are allowed or disallowed to access. It helps manage crawler activity, but it does not guarantee indexing or protect private information.
For WordPress website owners, understanding this file is especially useful when managing blogs, online tools, category pages, and large collections of URLs.
In this guide, you will learn what robots.txt does, how to create one in WordPress, which rules are useful, how to include your sitemap, and which mistakes can accidentally interfere with search engine crawling.
What Is a Robots.txt File?
A robots.txt file is a plain-text file that communicates crawling instructions to automated visitors, commonly called bots or crawlers.
Search engines use crawlers to discover website URLs and access page content. A website owner can use robots.txt to request that supported crawlers avoid certain areas of a website.
For example, a website might contain:
- Public blog articles.
- Online tools.
- Category and tag pages.
- Administrative areas.
- Internal search URLs.
- Other technical or non-public-facing paths.
Not every URL needs to be crawled equally. A robots.txt file can help guide crawlers toward the parts of a website that should receive attention.
The Robots Exclusion Protocol is documented in RFC 9309, which describes the standardized rules used by crawlers when interpreting robots.txt instructions.
What Does robots.txt Look Like?
A basic robots.txt file might look like this:
User-agent: *
Disallow:
This example identifies all crawlers with * and does not disallow any paths.
A more specific example could be:
User-agent: *
Disallow: /private-area/
Allow: /private-area/public-file.html
Sitemap: https://example.com/sitemap.xml
Here:
User-agentidentifies the crawler or group of crawlers.Disallowidentifies paths crawlers are requested not to access.Allowidentifies a path that can be accessed despite a broader disallow rule.Sitemapprovides the location of a sitemap.
The exact rules should reflect your website’s structure rather than being copied blindly from another website.
Why Does robots.txt Matter for SEO?
Robots.txt is part of technical SEO because it influences how supported crawlers access your website.
However, there is an important distinction: controlling crawling is not the same as controlling indexing.
A correctly configured file can help avoid unnecessary crawling of certain URLs, but an incorrect rule can prevent search engines from accessing pages that you actually want to appear in search results.
Consider a website that publishes 100 useful articles and online tools. If its robots.txt file accidentally blocks the entire website, Google may be unable to crawl those pages.
This does not mean every robots.txt change will improve rankings. Its purpose is to communicate crawling preferences, not directly increase a page’s ranking position.
For a wider understanding of website discoverability, read our guide on creating SEO-friendly URLs.
How to Find Your Existing robots.txt File
Before creating a new file, check whether your website already has one.
Open your browser and visit:
https://yourdomain.com/robots.txt
For example, if your website is example.com, the address would be:
https://example.com/robots.txt
If a robots.txt file exists, your browser should display its text content.
You might see rules such as:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
If you receive a 404 response, it may mean that no file is currently available at that address. It does not automatically mean your website has an SEO problem.
WordPress can generate a virtual robots.txt response when a physical file is not present. Some SEO plugins and hosting configurations may also affect how the file is managed.
Before making changes, identify whether your current robots.txt is a physical file or a WordPress-generated response.
How to Create a robots.txt File in WordPress
There are several ways to manage robots.txt on a WordPress website. The right method depends on your hosting setup, existing plugins, and level of technical experience.
Method 1: Use Your WordPress SEO Plugin
Some WordPress SEO plugins provide a robots.txt editing feature.
If your SEO plugin offers this option, you can usually find it within its general settings, tools, or file editor area.
A typical workflow is:
- Log in to your WordPress dashboard.
- Open your SEO plugin settings.
- Look for the robots.txt editor or file editor.
- Review the existing rules before changing anything.
- Add only the instructions your website actually needs.
- Save your changes.
- Open
/robots.txtin a new browser tab to verify the result.
If the plugin does not provide a robots.txt editor, use another supported method.
Method 2: Create the File Through Hosting File Manager
If you have access to your hosting account, you can create a physical robots.txt file directly.
Follow these steps:
- Sign in to your hosting control panel.
- Open File Manager.
- Navigate to your website’s document root, commonly named
public_html. - Check whether a file named
robots.txtalready exists. - If it does not exist, create a new text file.
- Name it exactly
robots.txt. - Add your required rules.
- Save the file.
- Visit
https://yourdomain.com/robots.txtto check the output.
The file should be located in the root of the website, not inside a WordPress theme folder or an individual plugin directory.
For example:
public_html/
wp-admin/
wp-content/
wp-includes/
robots.txt
wp-config.php
index.php
If a physical robots.txt file already exists, edit that file instead of creating a second copy.
Method 3: Use FTP or SFTP
Developers can also manage the file using an FTP or SFTP client.
Connect to the hosting server, navigate to the website’s root directory, and upload or edit robots.txt.
This method is useful when you manage multiple websites or prefer working with local files.
Be careful when editing production files. Keep a backup of the previous version so you can restore it if a rule causes unexpected crawling problems.
A Practical robots.txt Example for WordPress
A common starting point for a WordPress website is:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourdomain.com/sitemap.xml
Let’s understand what each instruction does.
User-agent: *
This applies the following rules to all crawlers that follow the standard robots.txt instructions.
Disallow: /wp-admin/
This requests that crawlers do not access URLs within the WordPress administration directory.
Allow: /wp-admin/admin-ajax.php
This creates an exception for the specified AJAX endpoint.
Sitemap:
This provides the URL of your XML sitemap, helping crawlers discover the URLs you have listed there.
This is a starting example, not a universal configuration. Check your website’s actual sitemap URL and technical requirements before using it.
Which WordPress URLs Should You Block?
Many website owners assume that every technical-looking URL should be blocked. That approach can create problems.
A better strategy is to understand what each URL does before adding a rule.
1. WordPress Administration Paths
The /wp-admin/ directory is commonly restricted for crawling.
However, WordPress websites may need the AJAX endpoint to remain accessible for certain front-end functions.
2. Private or Unnecessary Areas
If your website has specific areas that do not need crawler access, you can consider whether a disallow rule is appropriate.
Remember that robots.txt is publicly accessible. Anyone can read its instructions, so do not use it to hide sensitive directory names or confidential resources.
Use proper authentication and server-level access controls for private content.
3. Internal Search URLs
Some websites generate large numbers of search result URLs through query parameters.
For example:
https://example.com/?s=wordpress
Depending on the website’s structure and SEO strategy, you may decide to manage crawling of internal search URLs.
Do not add broad query-string blocking rules without testing their impact on important website pages.
4. URLs Generated by Plugins
WordPress plugins can create their own URL structures.
Before blocking any plugin-related path, check whether it contains resources needed to render your pages correctly.
Google needs access to important resources to understand how a page appears and functions.
What Should You Avoid Blocking?
One of the biggest robots.txt mistakes is blocking content or resources that search engines need to access.
Be careful with rules that block:
- Your entire website.
- Important blog categories.
- Public articles.
- Online tool pages.
- CSS and JavaScript files needed for rendering.
- Important images.
- Public pages that you want indexed.
For example, this rule is extremely broad:
User-agent: *
Disallow: /
It requests that all compliant crawlers avoid the entire website.
If you accidentally add it to a live website, search engines may be prevented from crawling important content.
Another mistake is blocking a page with robots.txt when the actual goal is to remove it from search results.
Google explains this distinction in its official robots.txt documentation.
robots.txt vs sitemap.xml: What Is the Difference?
Both files are used in technical SEO, but they serve different purposes.
| Feature | robots.txt | sitemap.xml |
|---|---|---|
| Main purpose | Communicates crawling rules | Lists important website URLs |
| Typical location | Website root | Website root or configured sitemap path |
| Contains | Crawler instructions | URL and optional page metadata |
| Helps with | Managing crawler access | Discovering website content |
| Guarantees indexing? | No | No |
Think of them this way:
A sitemap tells search engines which URLs you want them to discover.
A robots.txt file communicates which URLs crawlers are requested to access or avoid.
They complement each other, but one does not replace the other.
Google provides a separate explanation in its guide to XML sitemaps.
How to Add Your Sitemap to robots.txt
Adding a sitemap directive can make its location easier for supported crawlers to discover.
For example:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap_index.xml
If you use an SEO plugin that generates a sitemap index, use the correct sitemap index URL rather than assuming the file is named sitemap.xml.
For example, The Awais Ali uses this sitemap address:
https://theawaisali.com/sitemap_index.xml
The sitemap directive should point to the actual sitemap that your website publishes.
You can also submit your sitemap through Google Search Console. This gives you a way to monitor sitemap processing and identify reported issues.
Keep in mind that submitting a sitemap does not guarantee that every listed page will be crawled or indexed.
How to Check Whether Your robots.txt File Is Working
Saving the file is only the first step. You should verify that the server is delivering the correct content.
Use this checklist.
Step 1: Open the robots.txt URL
Visit:
https://yourdomain.com/robots.txt
Check that the file loads and displays the rules you intended to publish.
Step 2: Check the HTTP Response
The robots.txt URL should return an appropriate successful HTTP response when the file is available.
If it returns a 404, 403, or 500 response, investigate the server or file configuration.
Our guide to HTTP 401, 403, 404, and 500 errors explains what these responses mean and how to troubleshoot them.
Step 3: Review Google Search Console
Open Google Search Console and inspect important website URLs.
The URL Inspection tool can help you understand whether Google can access a page and what indexing information is available.
Check the affected page rather than assuming that a robots.txt file being visible in your browser means every page is crawlable.
Step 4: Check Important Pages
Review a selection of pages that should be available to search engines, such as:
- Your homepage.
- Important blog articles.
- Main tool pages.
- Important category pages.
Confirm that your rules do not unintentionally block their URLs or essential resources.
Step 5: Review After Every Major Change
If you change your WordPress theme, SEO plugin, website structure, or hosting configuration, review your robots.txt file again.
A rule that made sense for an older website structure may not be appropriate after a redesign.
Common robots.txt Mistakes That Can Affect Your Website
Here are some problems worth checking before publishing changes.
Blocking the Entire Website
A broad Disallow: / rule can prevent compliant search crawlers from accessing the entire site.
Always check the first lines of the file before publishing.
Using robots.txt to Hide Private Information
Robots.txt is not a security feature.
The file is publicly accessible, and its rules do not enforce authentication. Use passwords, permissions, or server-level restrictions to protect private content.
Blocking a Page Instead of Using Noindex
If your objective is to prevent a public page from appearing in Google Search, robots.txt is generally not the right solution.
Google needs to be able to crawl a page to see its noindex directive. If the page is blocked in robots.txt, the crawler may not be able to read that instruction.
Use an appropriate noindex directive when that is the intended outcome, and make sure the page is crawlable so search engines can process it.
Forgetting to Update the Sitemap URL
A website migration, SEO plugin change, or sitemap configuration update may change the sitemap address.
An outdated sitemap directive can point crawlers toward the wrong location.
Copying Rules from Another Website
A rule that works for one WordPress installation may be unsuitable for another.
Different plugins, themes, URL structures, and hosting configurations can require different settings.
Build your robots.txt file around your own website rather than copying a random example.
Frequently Asked Questions
What is robots.txt in WordPress?
Robots.txt is a text file used to communicate crawling instructions to search engine bots. In WordPress, it can be managed through a supported SEO plugin, a physical file in the website root, or a hosting file manager.
Where is the robots.txt file located in WordPress?
It is normally accessed through https://yourdomain.com/robots.txt. A physical file is usually stored in the website’s document root. WordPress can also generate a virtual robots.txt response when no physical file is present.
Does every WordPress website need a robots.txt file?
A robots.txt file is useful for communicating crawling preferences, but not every website needs custom rules. WordPress can provide a default virtual response, and unnecessary changes may introduce problems.
Does robots.txt improve Google rankings?
Robots.txt is not a direct ranking factor or a ranking shortcut. It helps communicate crawling instructions. Correct configuration can prevent accidental crawling restrictions, but rankings depend on many other factors.
Can robots.txt prevent a page from appearing in Google?
Not reliably. A blocked URL can still appear in search results if Google discovers it through other links. If you need to prevent a page from being indexed, use an appropriate noindex directive or password protection, depending on the situation.
What is the difference between Allow and Disallow?
Allow tells a supported crawler that a specified path can be accessed, while Disallow requests that it avoid a specified path. When rules overlap, search engines may apply their documented rule-selection logic.
Should I include my sitemap in robots.txt?
You can include your sitemap location using the Sitemap: directive. It is useful for communicating where your sitemap is located, but it does not guarantee that all listed URLs will be indexed.
Can I have more than one robots.txt file?
A website should publish one robots.txt file for a given protocol, host, and port combination. For example, the HTTP and HTTPS versions of a website are separate origins. Avoid creating conflicting files or assuming that multiple files will be combined.
Is robots.txt the same as a noindex tag?
No. Robots.txt communicates crawling instructions, while noindex is an indexing directive that tells search engines not to include a page in search results when they can crawl and process that directive.
Final Thoughts
A robots.txt file may contain only a few lines, but those instructions can influence how search engine crawlers access your website.
For WordPress owners, the safest approach is to inspect the existing file first, understand the purpose of every rule, keep important public pages accessible, and verify changes after publishing.
Do not add restrictions simply because a URL looks technical or unnecessary. A small configuration mistake can prevent crawlers from accessing content you actually want people to discover.
If you manage a website with multiple blog categories, online tools, and dynamic pages, treat robots.txt as part of your wider technical SEO setup, alongside a valid sitemap, descriptive URLs, and useful internal links.





