How Do You Manually Overwrite the robots.txt File in WordPress?

Introduction

If you’ve ever tried to control how search engines crawl your WordPress website, you’ve likely come across the robots.txt file. This small but powerful file plays a significant role in SEO and site visibility. But what happens when your plugins or theme settings limit what you can do? That’s when manual editing becomes your go-to solution.

Imagine you launch a new section of your site, but Google keeps skipping over it. You dig in and realize your robots.txt file is blocking those URLs. Now the only way to fix it is by manually editing the file – but how?

In this guide, you’ll learn step-by-step how to manually overwrite the robots.txt file in WordPress – whether you’re self-hosted, using a custom theme, or operating behind a plugin like Yoast. We’ll break it down with practical examples, safety tips, and best practices, so you stay in control of how search engines interact with your site.

Understanding the robots.txt File in WordPress

What is the robots.txt File?

The robots.txt file is a plain text file located at the root of your site (example.com/robots.txt). It tells search engine crawlers which parts of your site they’re allowed to crawl and index.

Example:

Why Edit It Manually?

Here are a few common reasons why you might want to manually overwrite it:

• Prevent indexing of staging or development pages.
• Allow search bots to crawl blocked CSS or JS files.
• Fix SEO issues flagged by Google Search Console.
• Enable crawling for specific bots while blocking others.

WordPress doesn’t generate a physical robots.txt file by default – it creates a virtual one, which means you need to take manual steps to create and control your own version.

Step-by-Step: How to Manually Overwrite robots.txt in WordPress

1. Create or Locate Your robots.txt File

If you don’t already have a physical robots.txt file, you need to create one.

Option A: Create via FTP or File Manager

1. Open your web hosting control panel (e.g., cPanel or Hostinger).
2. Navigate to the root directory of your WordPress installation (usually public_html/).
3. Create a new file named robots.txt.
4. Set permissions to 644 (readable and writable by the owner).

Option B: Use FTP Client

1. Open FileZilla (or any FTP client).
2. Connect to your server using your FTP credentials.
3. Go to the root folder.
4. Right-click and choose “Create New File,” then name it robots.txt.

2. Edit the robots.txt File

Once the file is created or located, you can open it and begin editing using a text editor.

Sample Syntax You Can Use:

To block all bots from a directory:

To allow Googlebot but block others:

To include your sitemap:

Sitemap: https://example.com/sitemap_index.xml

Important: Avoid blocking essential resources like /wp-content/themes/ or /wp-includes/ as it may affect your site’s appearance in search.

3. Disable Plugin Interference (If Needed)

SEO plugins like Yoast, Rank Math, or All in One SEO often manage the virtual robots.txt. If you want full control:

• Yoast SEO: Go to Tools > File Editor > Delete contents or disable virtual robots.txt
• Rank Math: Head to General Settings > Edit robots.txt and clear it out or disable overwrite

Alternatively, deactivate the plugin’s robots.txt management completely if they keep overriding your changes.

Common Use Cases and Examples

A. Allow Googlebot Only for Specific Folder

If you want Googlebot to access a media folder:

B. Block Bots from Crawling Search Pages

C. Prevent Crawling of Author Archives

Author pages can create duplicate content. To block them:

D. Allow Crawling of Important Scripts and Styles

Sometimes, your theme’s CSS or JS is blocked, affecting rendering. To fix:

Testing and Validating Your robots.txt File

After updating, it’s important to verify that it works correctly.

Use Google Search Console

1. Go to Google Search Console > URL Inspection.
2. Paste a page URL and click “Test Live URL”.
3. If access is denied, check what’s blocked in your robots.txt.

Use Online Tools

• robots.txt validator
• robots.txt testing tool

Both tools help test crawling rules for specific user-agents and paths.

Tips for Managing Your robots.txt File Safely

• Backup Before Changes: Always keep a backup of the previous version in case something breaks.
• Avoid Disallowing Entire Directories unless you’re sure they don’t contain content for SEO.
• Double-Check Syntax: One wrong rule can block search engines entirely.
• Limit Crawl Delay: Avoid using crawl-delay directives unless you experience performance issues.
• Use Allow Statements Wisely: Especially when you disallow a directory but want to make exceptions.

Conclusion

Manually editing your robots.txt file in WordPress gives you precise control over what bots can and cannot crawl. Whether you’re fixing SEO issues, blocking sensitive folders, or optimizing crawl budget, having a physical robots.txt file – and knowing how to manage it – can make a big difference.

Take a thoughtful, cautious approach. Small changes here can have a big impact on your site’s search performance. Always test your file after editing and keep learning about how different directives affect crawling behavior. By staying hands-on, you’ll avoid unnecessary indexing issues and maintain a clean, bot-friendly site.

FAQ Section

What is the purpose of a robots.txt file?

The robots.txt file tells search engine crawlers which parts of your website they are allowed or disallowed from accessing.

Where is the robots.txt file located in WordPress?

It is located at the root directory of your site, usually at yourdomain.com/robots.txt.

Can I create a robots.txt file manually in WordPress?

Yes, you can create it manually using FTP or your hosting control panel and place it in your website’s root directory.

How do I stop WordPress from creating a virtual robots.txt?

Disable the robots.txt feature in SEO plugins like Yoast or Rank Math to prevent virtual overrides.

What happens if I block all bots in robots.txt?

If you use Disallow: / for all bots, your site will be de-indexed from search engines, harming visibility.

Can I include my sitemap in the robots.txt file?

Yes, add a line like Sitemap: https://example.com/sitemap.xml to guide bots to your sitemap.

Do changes to robots.txt affect existing indexed pages?

Yes, blocking URLs can prevent recrawling and eventually cause them to drop from search results.

How can I test if my robots.txt file is working?

Use Google Search Console’s “URL Inspection” tool or online validators like robots.txt testers for confirmation.

Explore More WordPress Tips

How Do Hackers Find Admin Email Addresses on WordPress?

How Can I Manually Edit the DNS Zone in WordPress?

How to Hide the Admin Bar for Subscriber Users in WordPress?