How to Stop Spam Bots from Visiting Your Website

This past week, You Can Learn How to Code received a ton of referral traffic. I was initially excited when I logged into Google Analytics and saw the results. Upon closer examination, however, I discovered that I was receiving 50+ visits every day from the same website. Since all the traffic was only from one website where I do not have a deliberate inbound marketing strategy, it was clearly a red flag, and I had to take a couple of hours out of my afternoon last Friday to deal with it. If you have encountered the same situation, then I wrote this post specifically for you. I will discuss how to stop spam bots from visiting your website and ruining your bounce rate so that you can get back to doing what you love: blogging or developing websites. I will cover how you can accomplish this both in WordPress and in a website for which you can edit the htaccess file.

What is Referrer Spam and How Does It Work?

Referrer spam (also called referral spam) is a process through which a spammer accesses a website through a Google Analytics account. It works like this:

1. The spammer sends a number of fake visits to a website.
2. The website administrator logs into Google Analytics, sees the “referral”, gets curious about the referring website, and clicks on the link.
3. Once the website administrator clicks on the link, the spammer gains a visit to his website.

These spamming techniques are quite sophisticated. The spam bots visiting your website sometimes don’t actually visit your actual website. They bypass the process, sending a visit to Google Analytics.

There are SEO and marketing professionals who say that referral spam is harmless, but it is important to curb it for a couple of very important reasons.

Why is Referrer Spam Harmful?

Referrer spam is harmful for a couple of reasons:

1. It wreaks havoc on your bounce rate. A website that has a bounce rate of 100% means that all users go to just one webpage and leave. It is an indication that they are not engaging with your website on a deep level. If a bot is repeatedly visiting one page of your website and leaving right away, it is artificially inflating your bounce rate. A bot will most likely visit your homepage and then leave. If the homepage of your website serves to entice users to explore other areas of your site, then a high bounce rate is a bad thing. It means that visitors are not engaging with your content. So if your metrics have been overtaken by a spam bot, it also means that the engagement metrics you’re seeing for your website are artificially low.

2. It can inflate your metrics for international traffic. One of the last bots to visit my website sent referral spam from all over the world. When I logged in, I was very pleased to see all the traffic I had received that day from India, South America, and Europe. I was very excited until I discover that most of my international traffic that day was from referrer spam. I sat and watched my real time screen for a moment and actually saw the same bot ping my website from two different locations. This was, indeed, a sophisticated bot. After gazing at it in wonder for a few moments, I shook off my amazement and got to work blocking the offending websites.

How to Stop Referrer Spam Dead in Its Tracks

If you know that you have a problem with referrer spam, then you will like this next section because I am going to show you how you can block these harmful bots from visiting your website altogether.

Method 1: Plugin for WordPress Websites

If you maintain a WordPress website, there is a good plugin that you can use to block referrer spam. It is the “Block Referer Spam” plugin. I almost didn’t recommend it since “referrer” is not spelled correctly and it hasn’t been updated for one year. Other plugins I tried, though, failed to get rid of the referral spam, so I installed the “Block Referer Spam” plugin on one of the WordPress websites that I maintain, and I am happy to share that the plugin works.

Directions

If you’d like to install this plugin on a WordPress website that you maintain, you can follow these directions once you install the plugin. Once you install the plugin and follow these directions, you will be able to block the offending websites from visiting you.

1. Search for the “Block Referer Spam” plugin and install it.
2. Navigate to the Block Referer Spam screen. Leave the Auto Update and Block Mode default selections in place.
3. Go to your Google Analytics screen and review the web addresses for the referrals that you’re seeing.
4. Under the “Acquisitions” menu, select All Traffic and then Referral. If you’re getting traffic from Medium, Quora, YouTube or another social media channel, you will want to leave those alone. We are only interested in obtaining the web addresses from the spam bots so that we can add them to the plugin.
5. Carefully select the offending website by highlighting it with your mouse and copying the web address. Do not click on the link.
6. Navigate back to your Block Referer Spam plugin and paste the offending website into the Custom Blocks screen. This will update your htaccess file. (More on how that works below.)
7. Repeat this process until you have added all the offending websites to your Custom Blocks input area.
8. Once you have added all the websites that are sending you referral spam, click on “Save Changes” at the bottom of your Block Referer Spam screen.

The second method below covers how to edit the htaccess file yourself. The plugin does this for you automatically, so if you maintain a WordPress website and don’t feel comfortable editing the file, this first method is the best option.

Method 2: Edit the Htaccess File (for Advanced Users)

Typically, a web developer can add the offending website to a robot.txt file as a blocked URL, and that would prevent the spam bot from visiting the website. In more recent years, however, bots have gotten more sophisticated. They are now capable of completely overriding the robot.txt file and sending fake visits to Google Analytics without ever actually visiting a website. If this is enough to make your head spin, read on, and I will show how this happens since it is actually something you can see in Google Analytics.

Here is an example from this website You Can Learn How to Code. Around the middle of the morning on Friday, August 17, I became aware that a bot had been visiting my website repeatedly the last two days. As you can see, the top result is some odd variation of my home page. Until I realized what was going on, the bot hit my Google Analytics with 45 visits. The next result is my actual homepage, which demonstrates that the bot was sending fake visits.

Once I discovered the problem, I was able to edit my htaccess file to prevent these bots from actually visiting my website. I will show you line by line how I did it, but if you decide to go this route, please proceed very carefully. One wrong line or even character can bring down your entire website, so you must proceed with caution. I suggest downloading the file to your desktop and creating a backup copy. This way, if you run into problems, you can replace the problematic file with the original and try again.

In a separate article, I will show you how to eliminate the fake visits in Google Analytics. For now, we are creating a solution that will prevent the spam bots from accessing your website.

The most recent crawlers sent fake traffic from the following web addresses:

1. auto-seo-service.com (from August 17)
2. resell-seo-service.com (from August 16)

You can see the auto-seo-service.com web address as the top “referrer” in my Google Analytics results at that moment. Both of the auto-seo and resell-seo websites belong to Semalt, a company with known nefarious practices that has been sending referral spam since 2014 (or possibly longer). You can see these results in your own website in Google Analytics by clicking on Acquisition and Referrals.

To prevent these bots from accessing any real part of our websites, web developers can use a series of regular expressions. The following code prevents any Semalt, auto-seo-service, and resell-seo service website from visiting:

#BEGIN Referer Spam Blocker

RewriteEngine on

RewriteCond %{HTTP_REFERER} ^https?://[^/]+\.Semalt.com [NC,OR]

RewriteCond %{HTTP_REFERER} ^https?://[^/]+\.resell-seo-service.com [NC,OR]

RewriteCond %{HTTP_REFERER} ^https?://[^/]+\.auto-seo-service.com [NC,OR]

RewriteRule .* – [F]

#END Spam Blocker

There are some web developers that go one extra step and send the fake referral traffic back to the offending website, but I’d just assume not mess with potentially malicious bots. The [F] flag that you see near the end of the code states that these websites are completely forbidden, so you want to ensure that you don’t miss that last line.

Once you edit your file, you can upload it again to your website directory, and you will soon see an end to fake referral traffic. For me, the results were almost instantaneous. Within 30 minutes, the spam bots stopped visiting my website, and my analytics were well on their way towards getting restored to normal.

In a future post, I will also show you how to stop spam bots from manipulating your Google Analytics data. Even though this starts to get in the realm of marketing rather than web development, it is important that web developers understand how to implement solutions, especially ones that deal with regular expressions.

If you are dealing with referral spam, I hope this article has helped you. If you have any questions, please feel free to get in touch with me at laura.white@youcanlearnhowtocode.com.