Text
Understanding pagespeed tools and their pro's & cons
There are numerous solutions for measuring Core Web Vitals. But how do you select the right one? Learn about the differences between lab data tools like Lighthouse, field data tools like Pagespeed Insights, and real-time user monitoring like RUMvision.
What's the difference between lab- and field data?
We all want our websites to feel like they load fast and with minimum annoyance to our website visitors. That's because time and time again it's been proven there's a lot of value in improving Core Web Vitals.
To get insights into performance issues there are two ways of measuring it. Understanding the difference between doing synthetic/lab tests and measuring real users' experiences (field) is key if you want to make sure your optimization efforts are spent in ways that make a difference.
What is lab data?
Lab data is the name that the performance community uses for measuring performance in a controlled and synthetic environment. Or you know, a lab😉
It can mimic users' conditions - to a certain extent - and help you gain insights into what could happen in regards to web performance of your site if a user would visit it in its set conditions. It can be great to debug performance issues, as you can do a replay frame-by-frame, or signal issues that might arise in your code before the site goes live.
It is however not your real user, it's simulated and it can't account for a lot of factors a website visitors might experience, like interactivity, third party influences or conditions that might arise after a page has been loaded. For this you can utilize data gathered from your real users. A famous example of a lab data tooling is Google's Lighthouse, but there are other solutions like GTmetrix and Webpagetest that rely on lab testing. We'll look into their features, benefits and shortcomings later.
What is field data?
Field data is what’s actually used by Google’s Core Web Vitals. It measures the experiences of real website visitors - your real users. There are a lot of factors that cannot be measured in a controlled environment in a lab data test. That's why Google introduced the web-vitals and its metrics.
The great thing about field data, is that it captures the experience of all your real-world visitors who naturally all have different devices, network conditions and geographic locations. All these factors, and many more, can impact their experience and thus how your website's speed is perceived by your users. By taking into account fast as well as slow user experiences, field data provides a complete image of performance data from your users. Real user monitoring solutions, like RUMvision, also rely on field data to get you insights. They are in fact the solution to deepdive. But first, let’s discuss the Chrome User Experience Report (CrUX).
Because field data is based on real-user visits, it reflects the actual devices, network conditions, and geographic locations of your users. Field data is also commonly known as Real User Monitoring (RUM) data; the two terms are interchangeable.
Understanding CrUX, the base of Core Web Vitals
The low cost way to collect field data is by using Google’s free data: Chrome User Experience Report (CrUX). It's insights are limited, but mighty and defintely the best starting point when you have enough traffic on your website. CrUX data is also the data that is used for passing or failing the Core Web Vitals assessment.
Advantages
With CrUX data, you can get data from mobile and desktop traffic from the entire origin, homepage or the exact URL you enter. The nuance is that the exact URL must have enough data to be in the CrUX data set. For example, the blog overview page on web.dev has enough data. If you have a webshop with a lot of products then all the products together will have enough data, but individual product pages will not. Thus, it becomes difficult to see data from specific URLs with CrUX. There's no clear number of how many traffic a specific URL.
Here are some of the advantages of CrUX:
- It shows you where your real users experienced performance or UX issues, as it's based on field data.
- It shows you users experience over 6 key performance metrics (soon to be 5 when FID gets removed), that will help you discover where you should optimize.
- You find the 3 Core Web Vitals that are used for SEO ranking (LCP, INP, and CLS).
Core Web Vitals tool for historical data
Disadvantages
- As the name might suggest, CrUX only measures user experiences on browsers that run on Chromium. Experiences on Safari or Firefox are not part of this dataset.
- In fact, it only reports back after you've had enough data for Chrome visitors that month.
- The Core Web Vitals are shown aggregated over a 28 day period. That means that a code change or third parties impact only becomes fully visible after 28 days.
- It only shows the experience that people had on the 75th percentile. This means you’re missing insights into 25% percent of your visitors' experiences.
Although CrUX-based tools are a good starting point for investigating Web Vitals performance, we strongly recommend supplementing it with your own RUM.
RUMvision vs. Google Lighthouse
The biggest difference between Google Lighthouse and RUMvision is the fact that Lighthouse operates with lab data and RUMvision with field data. Therefore it measures 2 completely different things. We wrote a blog about "When to not use Google Lighthouse" to get a better understanding of its use-cases.
Often Lighthouse’s data comes from a device with slower internet than your own users, as such, it might not accurately represent the experience of your users.
Another difference is that Lighthouse does not take Time to First Byte (TTFB) into account. TTFB measures the time it takes between the first request for a source and when the first byte of that source begins to load. RUMvision does measure TTFB because it is one of the most noticeable factors for your users when loading takes a long time. These are the metrics that Google Lighthouse captures:
During a Lighthouse test the “user” does not interact with the page, this could introduce blind spots for metrics like the Cumulative Layout Shift (CLS) and Interaction to Next Paint (INP) that can happen at any given time. It only measures the first 5 seconds of a pageload. The closest Lighthouse has to offer for the INP is a Total Blocking Time, but the results will also differ from what real users experience.
If you want an idea of which script was actually harming during user interaction, with a button in example, the easiest way is to install RUM.
Apple.com score on Google Lighthouse
RUMvision vs. GTmetrix
Even though GTmetrix technically uses lab data they also try to simulate a real user's experience and behavior on a site. You can change these “user’s” variables like location, connection speed, etc. By doing this GTmetrix gives you the option to select variables of what you think represents your own audience. Keep in mind that this could introduce blindspots, as you won’t know which type of visitors experience bad core web vitals on your own website.
The GTmetrix score can also differ from other lab data tools like Google Lighthouse and WebPageTest, because all of them have a different methodology.
Report from GTmetrix of Apple.com
With RUMvision you can see which type of visitors you have on your website and how this correlates with web performance metrics. Not every user will be on a fast or slow device, but you have inbetweens as well. Therefore it is useful to capture visitors by the fact that they entered your website via an ad, are they first time visitors, what their internet speed is, and how fast their device is. On the visitor footprint of your website you can get these insights.
