How to Optimize the WordPress Robots.txt

Introduction

There are many ways to improve SEO for your website. One of the most effective ways is optimizing the Robots.txt file. As you know, the Robots.txt file is known as a powerful SEO tool because it will allow search engines to crawl your website. Therefore, you can freely boost the SEO of your site by optimizing the Robots.txt. In this blog today, we will introduce to you some methods for this topic clearly.

The Robots.txt file definition

This is a text file with the instructions that the webmasters or website owners will generate for search engine crawlers and indexes. This text file will be saved in the root directory of a domain. That means the crawlers will open it firstly when accessing your website.

You can see the basic format of a Robot.txt file below:

Let us explain to you the terms which are used in the Robots.txt file:

• User-agent: The name of the crawler
• Disallow: Prevents crawling of certain pages, directories, or files of a website
• Allow: Permit crawling of directories, files of pages
• Sitemap (optional): Display the location of the sitemap
• *: Stand for number or character
• $: Stand for the end of the line, you can use it to block web pages that contain a certain extension

In order to help you understand clearly, let’s see the following example:

In the Robot.txt example, we allow search bots to crawl files in the WordPress uploads folder.

On the other hand, we prevent search engines to access login, card, and fotos folders.

Finally, we give it the URL of our sitemap.

Why should you create a Robots.txt file for your WordPress site?

For a website, the search engines will also crawl and index all pages of it without a Robots.txt file. However, instead of crawling the folders or pages you want, they will crawl all content on your website.

There are 2 main reasons why you should generate a Robots.txt file for your site. First of all, with the help of it, you can manage all your content of your site easily. You can decide which pages or folders need to be crawled or indexed. That means the search bots won’t crawl the unnecessary pages, such as plugin files, admin pages or themes folders. Thus, it will assist your website to improve the index loading speed effectively.

The other reason is that you need to stop search bots from indexing an unnecessary post or page of your site. Therefore, it’s convenient and simple for you to hide a WordPress page or post from appearing in search results.

Methods to optimize the WordPress Robots.txt file

In this tutorial blog, we are going to bring you 2 ways to generate as well as optimize the Robots.txt in your WordPress site:

• Customizing Robots.txt file by using a plugin: All in One SEO
• Editing Robots.txt file manually with FTP

Using All in One SEO plugin to edit and optimize the Robots.txt file

In the WordPress market, there are many WordPress SEO Plugins supporting you in this issue. However, in this blog, we decide to guide you with All in One SEO – a popular plugin with over 2 million users.

First of all, let’s install it by going to Plugins -> Add New. Then search the name of plugin, install now and activate it.

After finishing some simple steps, you will return the admin dashboard. Now, you need to go to All in one SEO -> Tools.

In the Robots.txt Editor section, let’s switch the Enable Custome Robots.txt button to turn it on. So, you are able to create a custom Robots.txt file in WordPress after turning it on.

Let’s scroll down, you will see that this plugin show you a box with the Robots.txt preview. They are the defaul rules that WordPress added.

If you want to improve the SEO of your website, you can add more your own custom rules by clicking on the Add Rule button.

In order to add a new rule, let’s fill out the User Agent field with a * so that it will apply the rule to all user agents.

The next step, you just need to choose Allow or Disallow for the search bots to crawl.

After that, in the Directory Path, don’t forget to fill out the filename or directory path.

Last but not least, let’s pressing Save Changes button when you finish all your settings.

Now, all the new rule you have already added will be applied to the Robots.txt file automatically.

Utilizing FTP to customize and optimize Robots.txt file manually

This is another way to optimize the WordPress Robots.txt you may like. In order to to it, you have to use an FTP client to customize the Robots.txt file.

Firstly, let’s connect to your WordPress hosting account to use FPT client easily. When you access it, it’s simple for you to see the Robots.txt file in your site root folder. Let’s press the right mouse button and select View/Edit.

Conversely, if you can’t find it, you probably don’t have it. So, let’s create a Robots.txt file for your site first by clicking the right mouse button and choose Create new file.

Since the Robots.txt file is known as a plain text file, you are able to download it and customize via plain text editor, such as Notepad or TextEdit, etc. After that, you just need upload it back to your site’s root folder. It sounds simple, right?

Test the Robots.txt file of your WordPress site

When you have already created and set your Robots.txt file, it’s necessary for you to test it to find the errors and warn you. So, you need test it with a Robots.txt tester tool. We highly suggest you to try Google Search Console.

If you want to use Google Search Console, you make sure that your website is linked with this tool. After that, you are able to utilize the Robots Testing Tool by choose the proverty in the dropdown list. Now, it will add the Robots.txt file of your site automatically. If there is any error, it will warn you.

Wrap Up

All in all, the main purpose of optimizing the Robots.txt file is prevent the search bots to access pages that are unecessary to public, such as plugin folder or admin folder. Therefore, you don’t need to block WordPress tags, categories or archive pages to improve your SEO. In other words, you should follow the following Robots.txt format to generate an ideal Robots.txt file for your WordPress site.

We hope that the blog today may be useful for you. Don’t forget to share it with your friends and other webmasters. Furthermore, if you have any inquiry, let’s leave your comment below. So, we will reply on you as soon as posible. Now, it’s time for you to explore the ways how to optimize the WordPress Robots.txt file. See you in the next blogs.

5 / 5 ( 2 votes )

The post How to Optimize the WordPress Robots.txt appeared first on LTHEME.