If you've ever looked at your website analytics report, seen a traffic increase, and wondered if the visitors' data could possibly be true, you're not alone. Between human traffic, AI traffic, crawlers, and other nefarious actors, your traffic numbers may not be what they seem.
Bots aren't automatically the enemy
A bot is just software designed to perform tasks automatically online. They are typically used for repetitive things that would take humans a long time to complete.
When you hear the term “bot traffic”, you might assume you are dealing with malicious or unwanted behavior, but not all bots are bad. Google and other search engines use bots to crawl websites for information and indexing. Many companies use customer service bots to handle chat messages from visitors on their websites. Bots are everywhere, and most are just helpfully doing their job.
Here are some common bot types you might encounter and how they affect your website traffic:
- Chatbots - Answer user questions. AI assistants like ChatGPT, Claude, and Gemini act as chatbots, and they can send you traffic too. Read Is AI Changing How People Find Content? for more on this.
- Crawlers - Used to discover and index content. Search engines (AI and standard) use crawler bots to collect information and update their indices and systems. You might benefit from this if search engines send visitors to your site, depending on how popular your content is.
- Scrapers - Collect specific pieces of content, such as headlines, product details, or contact information for reuse or training. This can be seen in spiked pageview numbers or distorted metrics.
- Agents - Perform tasks such as filling out forms, making comparisons, or creating posts on social media. Agent bot traffic may resemble human traffic in your analytics.
Most websites will indicate what bot activity is permitted using a robots.txt file (typically found at yourwebsiteurl.com/robots.txt). This is just a plain text file stored on your website's server. A well-behaved bot will read the file, identify itself honestly by sending "User Agent" information with each request, and follow the rules allowed in the document.
What is malicious bot activity?
Malicious bots ignore those rules. They might disregard your robots.txt file entirely, violate your terms and conditions, or show up specifically to spam your servers, gather contact info, or attempt fraud.
Take a look at a few examples of user agent logs, and see if you can identify which is the real human and which are the bots.
| IP Address | Request | User Agent |
|---|---|---|
| 203.0.113.2 | GET /home | Chrome/126.0.0.0 Safari/537.36 |
| 198.51.100.20 | GET /home | SiteCheck-sitecrawl by Siteimprove.com |
| 31.57.184.84 | GET /home | Googlebot/2.1; +http://www.google.com/bot.html |
| 192.0.2.43 | GET /home | Chrome/126.0.0.0 Safari/537.36 |
The first row is an example of an actual human visiting the website.
The second two rows are bots designed to crawl websites for helpful information.
The last row is an example of a bot disguising itself as a human with a fake user agent.
Here's a look at the table again, this time with identifiers:
| IP Address | Request | User Agent | |
|---|---|---|---|
| Real human | 203.0.113.2 | GET /home | Chrome/126.0.0.0 Safari/537.36 |
| Bot | 198.51.100.20 | GET /home | SiteCheck-sitecrawl by Siteimprove.com |
| Bot | 31.57.184.84 | GET /home | Googlebot/2.1; +http://www.google.com/bot.html |
| Bot | 192.0.2.43 | GET /home | Chrome/126.0.0.0 Safari/537.36 |
When a bot identifies itself honestly, website analytics tools like Matomo can usually catch and segment it out of the data by looking at the user agent value. Unfortunately, if a bot is sending the same user agent value that a real human is, it becomes difficult to tell the difference, even for sophisticated analytics software. It is still possible to identify a badly behaved bot by looking at its IP address, even if it is lying about its user agent. However, we're seeing more attacks lately that use constantly changing IP addresses from the cloud, making them nearly impossible to block.
Look for these signs in your analytics report if you suspect malicious bot traffic. If you see all of these together, it's worth a closer look:
- An increase of over 100% in month-over-month pageviews (context matters - 10 pageviews jumping to 100 could just mean a post did well).
- Bots are mainly targeting the homepage URL, but it could be a sharp increase in pageviews to any specific webpage on your site.
- The increased traffic is direct traffic (acquisition not through search engine, social, or other website referral)
- The visitor log shows visits from various different countries with no real site interaction or indication they are a true visitor.
Any one of these on its own could have a perfectly normal explanation, so make sure you're reviewing carefully.
Do I need to worry about bots?
There's no foolproof way to block this kind of traffic, so don't worry about making a change to your website. Proxying traffic through services like Cloudflare can help, but they come at a cost that's not always justifiable for every site.
A simpler first step is to apply a geographic segment to your analytics before reviewing your data. The Matomo Analytics Dashboard for all ISU Sites already has a "United States" segment built in. So, if a particular month's numbers look strange, try applying it and see if the picture changes.
You don't need to catch every bot or chase down every spike in analytics. Just keep these patterns in mind when you have a question and consider the possibility of bot traffic when making decisions based on your website’s analytics.
The CALS/LAS Web Team continues to manage the analytics system to implement improvements and give you the most accurate metrics possible. Email websupport@iastate.edu if you have questions about your analytics report.