# Keeping the bots out - real users vs machines Traffic from AI, bots, and crawlers (ABCs) amounts to about 51% of all online traffic. For a company specifically tracking real users that raises a challenge. Who is fake? Who is real? And how do you filter that out? There is a word I see floating around on Reddit quite often called "Dead internet theory" - a theory where a lot of what we see online isn't real anymore, but just bots reacting to other bots. And not all bots are bad - some are actually very good for your business. But if you want to optimize for real users, who still are the main people who buy and read on your site, you need to reduce the noise in the dataset. Luckily, we are well equipped to handle it. Let me explain how we do it, and also why there might be interesting info for your company by actually keeping track of the bots - not just blindly blocking them, but classifying them. ## ABC is not as easy as 123 When we first started to build RUMvision five years ago, AI was not around as much. There was still a lot of bot traffic and crawlers, but it was manageable. I remember a while before the announcement of ChatGPT, we started to see a serious uptick in bot traffic. Little did we know how much that would grow in the years to come. ### Keeping out the bots Our approach to keeping out the bots over the last few years has been straightforward: - **AWS Cloudfront WAF:** We use Amazon's Web Application Firewall (WAF) to filter out the most well known ABCs at the edge, making sure they never even enter our dataset. - **Community blacklists:** We kept track of bot traffic in a blacklist that we - sometimes with help from the web performance community - kept up to date and fed back to our Cloudfront Edge to stop non-human traffic. This worked fine for a while. But the last year - and especially the last six months - non-human traffic has reached a new record. We've received same signals from CDNs. And the bots keep becoming smarter because they increasingly try to spoof real users. ## What are these ABCs anyway? To understand how they bypass traditional filters, we have to look at what makes up this non-human traffic: ### AI Think of tools like Perplexity, Google Gemini, OpenAI, Claude, and Mistral. They are all trying to "read" your website to gather information about your content and products. For business visibility, this is actually a good thing. With more people moving from traditional SEO to AI-led search, you want your site to show up in these results. ### Bots and runtime automation These are scripts trying to fake that they are humans. In an ideal world, you can identify bot traffic via their user agent string. But today, automated browsers leave traces in the browser runtime using frameworks like Playwright, Puppeteer, or WebDriver. They disguise their identity to scrape data, check inventories, or worse, launch spam and DDoS attacks. ### Crawlers These are often automated tools like SEO crawlers (Semrush, Ahrefs), uptime monitors, and search engine indexers like Googlebot or Bingbot. They are expected and necessary for your business, but they do not experience your site the way a human does. ## The bots in disguise Because bots are getting smarter, edge-level blocking is no longer enough. We started to recognize really strange traffic patterns that pass right through standard firewalls. For example, we noticed a site with a massive, unexpected spike in desktop traffic. When we did a deep dive into the data, it turned out all of them shared the exact same viewport dimensions (around a 10-inch screen size), which is not a viewport you would ever expect from a normal desktop user. That was highly suspicious. Other times, we see user strings claiming the visitor is using Safari, but the browser suddenly supports APIs that only exist in Chrome. It is a classic case of Chrome in disguise. ### Why traffic noise deserves its own layer This brings us to a crucial point: traditional bot protection and RUM classification are two entirely different things. Your CDN or firewall decides whether traffic should be *allowed* to hit your server. RUM classification is about understanding what kind of traffic *reached the browser* and produced measurements. If you block too aggressively at the edge, you might accidentally block useful crawlers, social media preview tools, or legitimate users. Instead of calling everything a bot and throwing the data away, we look at it as **traffic noise**. An in-app browser (like someone opening your link inside Facebook, Instagram, or LinkedIn) is not a bot. A strange viewport size is not a bot. But both can completely warp your Core Web Vitals data if they are mixed in with your normal visitors. ### The RUMvision honeypot to catch the noise We want to make sure you can separate human vs non-human data so you can analyze what your real visitors actually experience. To do this, we built a combination of a server-side and client-side honeypot. Instead of a simple yes-or-no filter, RUMvision classifies traffic by **noise levels** (none, low, medium, high, extreme) based on stable technical signals: - **Known bots:** Clearly identified search and AI crawlers. - **Runtime fingerprinting:** Detecting hidden automation traces like `navigator.webdriver` or Playwright scripts in the browser. - **Environment mismatches:** Flagging invalid viewport combinations and browser capability mismatches. - **In-app browsers:** Labeling traffic coming from social media apps so they don't skew your mobile performance metrics. ## Turning pollution into business visibility > **Tip:** Blindly deleting bot traffic means losing valuable business insights. Classification gives you the best of both worlds. You might wonder why a RUM solution should show this data instead of just deleting it? Filtering traffic noise answers different questions for different teams though. For example: ### For developers and UX teams When debugging Core Web Vitals, you only want to look at true "Human Visible Navigations" - real people looking at a visible browser tab. By filtering out high-noise traffic, you avoid chasing fake performance regressions caused by a crawler or an automated script hitting your checkout page. ### For SEO and marketing teams Instead of hiding the noise, you can inspect it. A traffic noise dashboard can show you exactly which pages AI bots are scanning, how often search crawlers visit your high-priority product templates, or how much traffic is driven by social media in-app browsers. Real-user data is still the best way to optimize your website, but it needs context. By treating traffic noise as an extra layer of insight, we keep your performance data clean while giving your business full visibility into how machines are interacting with your brand.