AhrefsBot is a web crawler that powers a link database for Ahrefs online marketing toolset. The bot constantly crawls the web to fill their database with new links and to check the statuses of the previously found ones. In other words, AhrefsBot is a bot that is primarily used as a backlink checker.
It is common to see AhrefsBot in your Google Analytics data especially when your site is generating a good amount of traffic and your competitors are planning your traffic/doing keyword research on your site.
In the next few monutes, you will learn everything about Ahrefs bot and how to easily block AhrefsBot.
Introduction to AhrefBot
What Is Ahrefsbot
According to ahrefs.com, every 24 hours the bot crawls over 6 billion web pages and updates Ahrefs’ index every 15 – 30 minutes.
Link data collected by Ahrefs is used for a variety of purposes and for digital marketers to plan, execute, and monitor online marketing campaigns.
Some people claim that Ahrefs’ web crawler is a bad bot that does not respect the robots.txt file and bypasses disallow and allow rules. However, Ahrefs disputes this claim by saying “We absolutely strictly respect robots.txt: both disallow and allow rules. Please see https://ahrefs.com/robot for full transparency on our crawler.”
The bot can also use up your website’s bandwidth and make your website slower. Another concern with the bot is that it is used by spammers that participate in referrer spam indexing in order to spam your website with unwanted referral traffic.
Other people have reported that the company behind the bot is known to be dishonest and provide misleading support to those who have questions about the bot but we cannot validate these claims.
How to block AhrefsBot quickly?
There are two ways you can block AhrefsBot. First, robots.txt and the second is .htaccess files. It is important to note that by blocking AhrefsBot, it will not negatively affect your ranking (if done right). However, it might lead to a reduced opportunity to be able to score some paid sponsored post opportunities.
More importantly, you want to block AhrefsBot because you want to avoid your competitors from doing research on your site.
Method #1: Update robots.txt file to block AhrefsBot
By default, the robots.txt file is a permission file that allow or disallow cralwers to crawl a website. By adding simple code to the file can stop bots such as AhrefsBot from crawling your site.
Here’s how it is done:
You are required to add this code to the robots.txt file. The robots.txt can be found in your website’s root folder. If you have problem searching for this, I recommend you to speak to your web hosting for some support.User-agent: AhrefsBot Disallow:/
Method #2: Block AhrefsBot using the .htaccess file
This method is a powerful and effective method to block other bots from crawling your website. Using .htaccess file, you can easily determine which bot to filter out from accessing your site. For this method to work, we will be blocking the IP ranges that AhrefsBot uses.
Add these details below to your .htaccess file. You can find your .htaccess file at the root of the files of your site.
deny from 173.199.64.0 – 173.199.127.255
Apart from that, here are other IP ranges that are often used by Ahrefs (and something you might consider):
54.36.148.0/2454.36.149.0/2454.36.150.0/24195.154.122.0/24195.154.123.0/24195.154.126.0/24195.154.127.0/24
It is important to understand that Ahrefs may change the IP’s from time to time. If that happens, simply add the same code to your .htaccess file with the updated IP addresses but keep this range in there.
Bonus: Blocking Ahrefs Bot
Here’s another nifty trick to block Ahrefsbot from scouting your site.
RewriteEngine On RewriteCond %{REQUEST_URI} !/robots.txt$ RewriteCond %{HTTP_USER_AGENT} ^.*AhrefsBot.*$ [NC,OR] RewriteRule ^.*.*
Summary:
I hope you find this tutorial on how to block AhrefsBot easy to follow and effective. If you need any help, feel free to reach out to me using the comment form below!