man working at computer sending spam bots to websites
| | | | |

How to Stop Spam Bots from Visiting Your Website

This past week, You Can Learn How to Code received a ton of referral traffic. I was initially excited when I logged into Google Analytics and saw the results. Upon closer examination, however, I discovered that I was receiving 50+ visits every day from the same website. Since all the traffic was only from one website where I do not have a deliberate inbound marketing strategy, it was clearly a red flag, and I had to take a couple of hours out of my afternoon last Friday to deal with it. If you have encountered the same situation, then I wrote this post specifically for you. I will discuss how to stop spam bots from visiting your website and ruining your bounce rate so that you can get back to doing what you love: blogging or developing websites. I will cover how you can accomplish this both in WordPress and in a website for which you can edit the htaccess file.

What is Referrer Spam and How Does It Work?


Referrer spam (also called referral spam) is a process through which a spammer accesses a website through a Google Analytics account. It works like this:

  1. The spammer sends a number of fake visits to a website.
  2. The website administrator logs into Google Analytics, sees the “referral”, gets curious about the referring website, and clicks on the link.
  3. Once the website administrator clicks on the link, the spammer gains a visit to his website.

These spamming techniques are quite sophisticated. The spam bots visiting your website sometimes don’t actually visit your actual website. They bypass the process, sending a visit to Google Analytics.

There are SEO and marketing professionals who say that referral spam is harmless, but it is important to curb it for a couple of very important reasons.

Why is Referrer Spam Harmful?

Referrer spam is harmful for a couple of reasons:

1. It wreaks havoc on your bounce rate. A website that has a bounce rate of 100% means that all users go to just one webpage and leave. It is an indication that they are not engaging with your website on a deep level. If a bot is repeatedly visiting one page of your website and leaving right away, it is artificially inflating your bounce rate. A bot will most likely visit your homepage and then leave. If the homepage of your website serves to entice users to explore other areas of your site, then a high bounce rate is a bad thing. It means that visitors are not engaging with your content. So if your metrics have been overtaken by a spam bot, it also means that the engagement metrics you’re seeing for your website are artificially low.

2. It can inflate your metrics for international traffic. One of the last bots to visit my website sent referral spam from all over the world. When I logged in, I was very pleased to see all the traffic I had received that day from India, South America, and Europe. I was very excited until I discover that most of my international traffic that day was from referrer spam. I sat and watched my real time screen for a moment and actually saw the same bot ping my website from two different locations. This was, indeed, a sophisticated bot. After gazing at it in wonder for a few moments, I shook off my amazement and got to work blocking the offending websites.

How to Stop Referrer Spam Dead in Its Tracks

If you know that you have a problem with referrer spam, then you will like this next section because I am going to show you how you can block these harmful bots from visiting your website altogether.

Method 1: Plugin for WordPress Websites

image of the block referer spam pluginIf you maintain a WordPress website, there is a good plugin that you can use to block referrer spam. It is the “Block Referer Spam” plugin. I almost didn’t recommend it since “referrer” is not spelled correctly and it hasn’t been updated for one year. Other plugins I tried, though, failed to get rid of the referral spam, so I installed the “Block Referer Spam” plugin on one of the WordPress websites that I maintain, and I am happy to share that the plugin works.

Directions

If you’d like to install this plugin on a WordPress website that you maintain, you can follow these directions once you install the plugin. Once you install the plugin and follow these directions, you will be able to block the offending websites from visiting you.

  1. Search for the “Block Referer Spam” plugin and install it.
  2. Navigate to the Block Referer Spam screen. Leave the Auto Update and Block Mode default selections in place.
  3. Go to your Google Analytics screen and review the web addresses for the referrals that you’re seeing.
  4. Under the “Acquisitions” menu, select All Traffic and then Referral. If you’re getting traffic from Medium, Quora, YouTube or another social media channel, you will want to leave those alone. We are only interested in obtaining the web addresses from the spam bots so that we can add them to the plugin.
  5. Carefully select the offending website by highlighting it with your mouse and copying the web address. Do not click on the link.
  6. Navigate back to your Block Referer Spam plugin and paste the offending website into the Custom Blocks screen. This will update your htaccess file. (More on how that works below.)
  7. Repeat this process until you have added all the offending websites to your Custom Blocks input area.
  8. Once you have added all the websites that are sending you referral spam, click on “Save Changes” at the bottom of your Block Referer Spam screen.

screen shot of the block referer spam plugin

The second method below covers how to edit the htaccess file yourself. The plugin does this for you automatically, so if you maintain a WordPress website and don’t feel comfortable editing the file, this first method is the best option.

Method 2: Edit the Htaccess File (for Advanced Users)

Typically, a web developer can add the offending website to a robot.txt file as a blocked URL, and that would prevent the spam bot from visiting the website. In more recent years, however, bots have gotten more sophisticated. They are now capable of completely overriding the robot.txt file and sending fake visits to Google Analytics without ever actually visiting a website. If this is enough to make your head spin, read on, and I will show how this happens since it is actually something you can see in Google Analytics.

Here is an example from this website You Can Learn How to Code. Around the middle of the morning on Friday, August 17, I became aware that a bot had been visiting my website repeatedly the last two days. As you can see, the top result is some odd variation of my home page. Until I realized what was going on, the bot hit my Google Analytics with 45 visits. The next result is my actual homepage, which demonstrates that the bot was sending fake visits.

wrong homepage address

Once I discovered the problem, I was able to edit my htaccess file to prevent these bots from actually visiting my website. I will show you line by line how I did it, but if you decide to go this route, please proceed very carefully. One wrong line or even character can bring down your entire website, so you must proceed with caution. I suggest downloading the file to your desktop and creating a backup copy. This way, if you run into problems, you can replace the problematic file with the original and try again.

In a separate article, I will show you how to eliminate the fake visits in Google Analytics. For now, we are creating a solution that will prevent the spam bots from accessing your website.

The most recent crawlers sent fake traffic from the following web addresses:

  1. auto-seo-service.com (from August 17)
  2. resell-seo-service.com (from August 16)

referrer spam from auto-seo-service.com at top of listYou can see the auto-seo-service.com web address as the top “referrer” in my Google Analytics results at that moment. Both of the auto-seo and resell-seo websites belong to Semalt, a company with known nefarious practices that has been sending referral spam since 2014 (or possibly longer). You can see these results in your own website in Google Analytics by clicking on Acquisition and Referrals.

To prevent these bots from accessing any real part of our websites, web developers can use a series of regular expressions. The following code prevents any Semalt, auto-seo-service, and resell-seo service website from visiting:

#BEGIN Referer Spam Blocker

RewriteEngine on

RewriteCond %{HTTP_REFERER} ^https?://[^/]+\.Semalt.com [NC,OR]

RewriteCond %{HTTP_REFERER} ^https?://[^/]+\.resell-seo-service.com [NC,OR]

RewriteCond %{HTTP_REFERER} ^https?://[^/]+\.auto-seo-service.com [NC,OR]

RewriteRule .* – [F]

#END Spam Blocker

There are some web developers that go one extra step and send the fake referral traffic back to the offending website, but I’d just assume not mess with potentially malicious bots. The [F] flag that you see near the end of the code states that these websites are completely forbidden, so you want to ensure that you don’t miss that last line.

Once you edit your file, you can upload it again to your website directory, and you will soon see an end to fake referral traffic. For me, the results were almost instantaneous. Within 30 minutes, the spam bots stopped visiting my website, and my analytics were well on their way towards getting restored to normal.

In a future post, I will also show you how to stop spam bots from manipulating your Google Analytics data. Even though this starts to get in the realm of marketing rather than web development, it is important that web developers understand how to implement solutions, especially ones that deal with regular expressions.

If you are dealing with referral spam, I hope this article has helped you. If you have any questions, please feel free to get in touch with me at laura.white@youcanlearnhowtocode.com.

Similar Posts

9 Comments

  1. This article is helpful. I now understand what a spam bot is. I had an experience with them before when I created my first blog. I used to see visits to my site that didn’t last longer than one second. Also, I used to get a lot of comments awaiting approval which came with attached links to some websites. The comments did not relate to the post I published, and eventually, my AdSense was blocked by Google because I used to get a lot of traffic from an unknown source. This article has really made me understand it all. I enjoyed reading this and will install the plugin you recommend.

    1. Thanks, Matron. I’m glad this article helped you. The spam bots sometimes visit for one second. Other times they spend more time on your website, but they rarely visit more than one webpage. If you maintain a WordPress website, the Block Referer Spam plugin will help.

  2. It hard enough to keep your website running, we have also to fight spam bots from visiting my website. I am glad to read this article though I have not started using google Analytics I was waiting to have a lot of content on my websit.

    I am interested in the Block Referer Spam plugin, and I will check it out.

    Thank you for this article very informative and helpful but also eye opening.

  3. Laura,
    Your article has been a sigh of relief. I had been searching on the internet for the past two days to find a way to stop click bots from visiting my website(For past couple of days my website was facing spam/referral/click bots attack).
    I read about various technique some of them too technical for me to execute. I am really happy to stumble upon your post. Installed the plugin sometime back and it seemed to have worked(too early to say).
    Really thankful to you!
    I owe you a treat!
    warm regards,
    Yajiv

    1. I am so glad this article was helpful to you. I received word from a reader that the plug in no longer works. I ended up adding regular expressions in Google Analytics to get rid of the worst spam bots and it worked well. I will record a video on how to do this soon.

  4. Hello Laura,

    Great article.

    I am facing the same problem from yesterday and looking for a solution.

    The plugin that you suggested Block Referer Spam is not compatible with the latest WordPress version. This message shows on their sales page:

    This plugin hasn’t been tested with the latest 3 major releases of WordPress. It may no longer be maintained or supported and may have compatibility issues when used with more recent versions of WordPress.

    Is there any other good plugin?
    Can these referrer spam bots hurt my site ranking?

    Please help me out.

    1. I am sorry to hear you’re struggling with referral spam, and thank you for letting me know that the plug in is no longer working. I will look into updating the article.

      I ended up enacting a rather technical solution that involves adding regular expressions in Google Analytics to filter out the specific website addresses. It has done the trick for me. Explaining the solution requires more detail than I can go into in a comment, so I’ll record a video about this soon and will let you know when it’s up.

Leave a Reply

Your email address will not be published. Required fields are marked *