SEO Agency
SEA Agency
Grow your business quickly with paid search (SEA).
Social ads
The agency
SEO Agency
SEA Agency
Grow your business quickly with paid search (SEA).
Social ads
The agency
The robots.txt file, also known as the robots exclusion protocol, is used to prevent search engine crawlers from accessing certain files on your site. It acts as a guide for search engine bots, whether Googlebots, Bingbots, or Yandex bots, in crawling your site, by blocking them from accessing certain URLs. This is a file that you must include in your site if you want it to rank.
This file is placed at the top of a website. It is therefore one of the first files analysed by the crawlers. To check that it is there, simply type the address of your site in the search bar and add “/robots.txt” after it. If it is not there, a 404 error will be displayed.
Quick Access
This file makes it possible to prevent crawling, and therefore indexing:
In addition, the robots.txt file tells search engine crawlers about your sitemap, so they can easily find it.
The robots.txt file, if used properly, can help the SEO of your website. Indeed, it allows :
It is a file that is used extensively, in particular for e-commerce sites, as it allows the problems of duplicate content arising from faceted searches to be resolved. This is a navigation method found on many e-commerce sites, which allows users to quickly find what they are looking for by filtering the different products offered on the site. However, such a search method leads to the creation of many pages with very similar content, because of the multitude of possible combinations between the different filters and categories. These pages then risk cannibalising each other, as well as diluting the PageRank captured by your strategic pages.
This is where the robot.txt file comes into play, which you can use to prevent these pages from being crawled by search engine robots, while leaving them accessible to users.
This robot.txt file is also used to prevent Google from crawling some specific pages on your website:
This is because these are pages that are generally used to attract leads. For example, if you want to obtain users’ details before giving them access to these documents, the robot.txt file allows you to block access to anyone who has not filled in the required information.
As a webmaster, there are probably some parts of your site that you want to keep private, such as certain personal files, or URL parameters.
Finally, this robot.txt file can be used to indicate a crawl delay to avoid your servers being overloaded by search engine crawlers. Indeed, when the crawlers explore several contents of your site at the same time, it can cause an overload of your servers which do not have the capacity to load so much content simultaneously.
The robots.txt file is either manually created or automatically generated by most CMS such as WordPress, and must be found at the root of a site. But you can also use online tools to create this file.
If you want to create your robots.txt file manually, you can use any text editor following certain rules:
Be careful, your robots.txt file must not be larger than 512 Kb.
Creating your robots.txt file is very simple. To do so, you just have to go to the “general settings” tab of your plugin, then click on “edit robots.txt“. You can then write your file directly in the plugin, which will automatically integrate it into your site.
Please note that if you had already added a robots.txt file to your site before installing the Rank Math plugin, and you now want to manage it through the plugin, you must delete the file from your site before you can edit it in Rank Math.
You can also use the Yoast plugin to manage your robots.txt file. As with Rank Math, the way it works is quite simple. Go to the “referencing” menu of the plugin. Then click on “tools“, then on “edit file“. If you have not yet added the file to the plugin, you just have to create it by clicking on the associated button. All you have to do is make the desired changes before clicking on “save“.
The robots.txt file uses a specific language in which there are a few regular expressions, called Regex, that allow you to simplify writing the robots.txt. Here are some of the common expressions.
This command allows you to use specific search engines. The robot.txt file is the first file that search engine robots scan. They then check that they have been mentioned in this file. If they see their name appear, they will then read the commands that have been assigned to them.
To mention a search engine, you simply insert its name after the User-agent command. For example, if you want to mention Google, you would write “User-agent: Googlebot“. Furthermore, if you wish to centralise all the commands addressed to the search engines by identifying them all at once, you only need to write the following command: “User-agent: * “.
This command prevents crawlers from crawling certain parts of your website. However, you can only add one command per line, which is why there are multiple lines of “disallow” commands in a row in robots.txt files.
For example, if you put “Disallow: */catalog/” after your “User-agent: Googlebot” command, you are disallowing Google’s robots to visit all of your catalogue pages.
This command only applies to a Google crawler, called Googlebot, and gives it access to a page or sub-folder, even if it is denied access to its parent page.
For example, if you add the command “Allow: /wp-admin/admin-ajax.php” after the command “Disallow: /wp-admin/“, you allow the Googlebot to access the sub-folder “admin-ajax.php“, but not to crawl your “wp-admin” page.
The Crawl-Delay command allows you to ask crawlers to wait a few seconds before crawling your site. For example, by entering “Crawl-Delay: 20“, you ask the robots of the search engine(s) concerned to wait 20 seconds before entering your site.
As you can imagine, this command allows you to directly indicate your sitemap to the crawlers. To do this, you simply insert the URL of your sitemap after the “Sitemap:” command.
There are some syntax elements specific to robots.txt files that are important to know:
To optimise your robots.txt file, it is important to adopt some good practices:
To ensure that all your important URLs are indexable by Google, you can test your robots.txt file. To do this, simply register your site with Google Search Console, then click on Crawl in the menu, then click on Robots.txt Testing Tool.
However, this robots.txt file has some limitations in its utility. These limitations are as follows:
SERP
H1 Tag
trust flow
seo data
popularity index
link juice
alt attribute
semantic cocoon
meta description
internal mesh
robots.txt
duplicate content
Boost your Visibility
Do not hesitate to contact us for a free personalised quote
