Content:

Get a free consultation from an expert on your project

Secrets of Robots.txt

Robots.txt is an important tool for SEO site optimization. His task is to establish contact with the bots of search engines and inform them about which folders they need to look into to scan data and which ones need to be bypassed.

It should be said right away that no perfect robot file.txt for WordPress, which would suit all sites on this CMS without exception. But there are basic things that can be adjusted depending on which pages of the site you want to open or close for search engines.


What is needed to be indicated in robots.txt and how to create such a file? To begin with, you should understand which commands it contains and how it “speaks” with the bots of search engines.

The main components of the robots file for WordPress are directives and variable data indicating the file (file group). 

Types of directives and features of their filling

User-agent - calls the name of the search robot to which the commands are addressed: Googlebot (Googlebot-news, Googlebot-image, etc.), Yandexbot (YandexImages, YandexVideo, etc.), Bing, Slurp, Mail.ru. To handle all bots, the * (User-agent: *) symbol is used. This is the main robot file directive for WordPress, it should be at the very beginning of the file.


Allow and Disallow - opens and, accordingly, closes access to files and pages of the site. It is worth immediately making a reservation that the teams in robots.txt do not force search robots to perform, they rather highly recommend. At the same time, bots have the right to behave as they please, and can even scan files that are closed from their eyes.


For communication with bots in the robot.txt file, apply special operators:

  • * - any characters before and after data variables. 
  • $ - as a point at the end of the sentence, this symbol shows that the sign in front of it is the last in the address bar;
  • # - means that the author’s comments are followed and they are not taken into account by the PS;
  • / - shows which sections and pages the indication belongs to.

Sitemap - indicates the path to the XML file of the site map. Usually indicated at the end of the robots.txt file. The standard view of the link to the site map is https://domen.com/sitemap.xml, where domen.com is the domain name of your site on WordPress. Sitemap.xml - name of the site map file.


Host - Previously used by Yandex search robots to determine the main mirror of the site. Now this directive has lost its relevance and the Robots do not need to indicate it. Robots Google host has never been counted.


Crawl-delay - sets the time interval between the scanning processes. It is taken into account by Yandex, Mail.ru, Bing and Yahoo! A second is taken as the unit of account, it can be indicated by an integer or as a decimal fraction with a dot as a separator: 1.0, 0.7, etc. 


Clean param - shows Yandex bots dynamic addresses with a question mark that must be hidden in order to avoid duplication of content. 

How to make Robots for Wordpress: step-by-step instructions for filling out

If you want to specify different conditions for the Yandex and Google robots, then you should prescribe directives with commands for each of these search engines separately. If you don’t pursue such a goal, you can use the general rules, which means that * must be indicated in the User-agent field.

Правильный robots.txt для wordpress

The algorithm for writing the correct document for WordPress robots.txt:

  • Indicate the addressee of commands (User-agent)
  • Close from indexing (Disallow):
    - admin files;
    - personal cabinets, registration and authorization forms;
    - tools for working with orders (basket, forms for filling data, etc.);
    - approaches to interactive user interfaces ;
    - folder cgi;
    - data on the search function;
    - service pages;
    duplicate pages and sections;
    - data on filtering parameters, comparisons and sort rules;
    UTM tags;
    - plugins and graphics of topics of design.
  • We open (Allow) those files and documents that are necessary to index, but they are inside already closed categories, for example, images and JavaScript
  • Enter site map data (Sitemap)
  • If necessary, we prescribe Crawl-Delay
  • If necessary, we prescribe Clean-Param (considered only for Yandex bots)
  • Through Google and Yandex services, check the correct filling of robots.txt


It seems to be nothing complicated, but quite often many questions arise as to which files and folders should be closed from scanning, and which are desirable to index. As part of the SEO Market online course, we dwell on this topic in detail, and as a bonus our experts will tell you, as in your case, it is better to compile a robots.txt file for your site on WordPress and check its completion.

Get a free consultation from an expert on your project

Get a free consultation from an SEO expert on your site

How to create a robots.txt file in WordPress

There are several ways to create robots.txt for WordPress - manually and using plugins. Consider them in more detail.

Creating a robot.txt document using text editors

Any standard text editor is applicable to write robots.txt, the same notebook will do. After all the data is entered into the document, save it on a computer under the name robots.txt. It remains to place it on the WordPress server.

We go to the FTP server of the site, find the public_html folder of the main directory and drag our robots.txt file into it. That's it, search bots see recommendations for indexing.

Как добавить robots.txt в WordPress

Creating robots.txt using the WordPress plugin

Consider immediately the two most popular plugins for WP - Yoast SEO and All in One SEO Pack

  1. Yoast SEO
    After installing and activating the plugin, go to the SEO tab Tools section and select File Editor, then click Create Robots.txt file. On the page that opens, you can compile a list of commands for the bots of search engines or edit existing data. After checking the correct filling, save the changes.
     
  2. All in One SEO Pack
    Upon completion of the All in One plugin for WordPress, in order to create the robots.txt file, go to the Module Management section, select the robots.txt icon and press Activate. In the opened form, we introduce additional rules or delete (do it inactive) previously added commands. The editor independently places the directories in places, you do not need to enter the name of the directories.

    A distinctive feature of All in One SEO Pack is the customizable blocking of malicious bots that overload the site without bringing you any benefit.

Conclusion

Regardless of the selected way to create the robots.txt file, if all directories and commands are entered correctly, search bots will see him and in most cases follow the specified recommendations. After downloading the file, we recommend that you check it using the Yandex.Webmaster and Google Search Console services. Please note that when checking on Google, errors will be indicated in specific commands for Yandex or other systems that he does not understand. Do not pay attention to it.


Now you know how to make the robots.txt file for WordPress and you can direct the feet of search robots in the direction you need, relieving the load on the site and optimizing the process of indexing data on the pages of your site.