There are numerous solutions for measuring Core Web Vitals. But how do you select the right one? Learn about the differences between lab data tools like Lighthouse, field data tools like Pagespeed Insights, and real-time user monitoring like RUMvision.
Did you know we use different methods to track web performance?
Everyone wants a site that is fast. It is better for conversions, user happiness, SEO, PPC, and AI. But to improve your Core Web Vitals, you need to understand that not all tools measure the same.
For instance, did you know that the score on the left in this meme (Lighthouse score) vs the score on the right (Core Web Vitals passed/failed) are measured in completely different ways? Let's figure out how it works, and how RUMvision factors in.
What is lab data?
Lab data is the name that the performance community uses for measuring performance in a controlled and synthetic environment. Or you know, a lab 😉
For this you can utilize data gathered from your real users. A famous example of a lab data tooling is Google's Lighthouse, but there are other solutions like GTmetrix and Webpagetest that rely on lab testing. We'll look into their features, benefits and shortcomings later.
It can mimic users' conditions to a certain extent and help you gain insights into what could happen in regards to web performance of your site if a user would visit it in its set conditions. It can be great to debug performance issues, as you can do a replay frame-by-frame, or signal issues that might arise in your code before the site goes live.
It is however not your real user. It is simulated and it cannot account for a lot of factors a website visitor might experience, like interactivity, third-party influences or conditions that might arise after a page has been loaded.
What is field data?
Field data is what is actually used by Google's Core Web Vitals. It measures the experiences of real website visitors, your real users. There are a lot of factors that cannot be measured in a controlled environment in a lab data test. That is why Google introduced the web-vitals and its metrics.
The great thing about field data is that it captures the experience of all your real-world visitors who naturally all have different devices, network conditions and geographic locations. All these factors, and many more, can impact their experience and thus how your website's speed is perceived by your users. By taking into account fast as well as slow user experiences, field data provides a complete image of performance data from your users.
Because field data is based on real-user visits, it reflects the actual devices, network conditions, and geographic locations of your users. Field data is also commonly known as Real User Monitoring (RUM) data; the two terms are interchangeable.
Real user monitoring solutions, like RUMvision, rely on field data to get you insights. They are in fact the solution to deep dive. But first, let's discuss the Chrome User Experience Report (CrUX).
When to use lab or field data
Knowing when to deploy the right tool is valuable. That way, it can save you a lot of time and headaches. My own rule is to always use lab data supplementary to RUM data when a website is live.
Example of how to use RUM and lab data together
Suppose a website has a high Interaction to Next Paint (INP). I can see this in CrUX data. However, I then have no idea why my INP is on the high side. This is what I use RUM data as well as attribution and, of course, the LoAF API for.
Our free CrUX tool might tell you that your site has a poor INP. But it will not tell you if the culprit is your mobile menu, your cookie banner, or your "Add to Cart" button. RUM data captures the exact HTML element the user clicked, allowing you to stop guessing and start fixing.
Now I know which elements people actually interact with and thus produce high latency. Now that I have this data, I can use DevTools to create a performance trace and see what exactly is happening.
Also read: Using your RUM LoAF data for INP debugging purposes
Staging environment
When a website is under development, the value of RUM data sinks because there are only developers working on your website. These often work on desktop under fast internet connections, for example wired internet in the office. On a staging environment, it is useful to stick to best practices, for example those Google Lighthouse shows you. In addition, it is never a bad idea to hire a consultant to do a quick scan on your staging environment to pick out UX deteriorating factors before going live.
The minute your website goes live, it is useful to deploy a RUM tool. This is because in the beginning, there are still many issues that come to light and need to be fixed. When you depend on CrUX data here, you have to wait until there is enough data and you are behind the times because of aggregated data.
Search ranking (SEO) purpose
When your approach is to engage with the Core Web Vitals to improve SEO, it is good to know that you are always and only looking at field and RUM data, for example through PageSpeed Insights or Search Console.
An 85 score or even a 100 score on Lighthouse is not going to contribute to better rankings. To get SEO benefits from Core Web Vitals you need to pass:
- Largest Contentful Paint (LCP)
- Interaction to Next Paint (INP)
- Cumulative Layout Shift (CLS)
Different Core Web Vitals tools
There are lots of different types of tools to measure your Core Web Vitals with. The important thing to grasp is that each tool can use three types of data:
- Lab data (synthetic)
- CrUX data (real users, but delayed and not detailed)
- RUM data (real users, real-time, and able to deep dive)
RUMvision vs. CrUX
The low-cost way to collect field data is by using Google's free data: the Chrome User Experience Report (CrUX). Its insights are limited, but mighty, and definitely the best starting point when you have enough traffic on your website. CrUX data is also the data used for passing or failing the Core Web Vitals assessment.
With CrUX data, you get data from mobile and desktop traffic from the entire origin, homepage, or the exact URL you enter. The nuance is that the exact URL must have enough data to be in the CrUX data set.
Core Web Vitals tool for historical data
Understanding the 28-day rolling window
A frequent point of confusion is how CrUX aggregates its data. It is important to distinguish between the aggregation window and the publishing frequency. CrUX always calculates its metrics based on a 28-day rolling window. Whether you look at daily, weekly, or monthly reports, the underlying data always spans the previous 28 days.
Depending on the tool you use, you are accessing different publishing frequencies of that same 28-day period:
- CrUX API: Updates daily and runs about two days behind.
- History API: Updates weekly and provides overlapping 28-day periods for historical tracking.
- BigQuery: Updates monthly. This is also the only dataset that allows you to filter data by country.
- PageSpeed Insights and Search Console: These tools pull directly from the API.
Why CrUX data takes weeks to update
When you deploy a performance improvement, you will not see an immediate change in your CrUX data. This is not because the data is arriving late - the data itself is usually only two days old.
The delay happens because of how the 28-day window handles the 75th percentile. Each day, the window drops the oldest day of data and adds the newest day. In the first week after a fix, your new, faster sessions only make up a small fraction of the total 28-day dataset. Because CrUX measures the 75th percentile, the overall score will not improve significantly until your new, faster sessions make up the majority of the data. Typically, it takes two to three weeks for the new data to cross that threshold and reflect the actual improvement.
If you just used our free CrUX checker hoping to validate yesterday's release, this rolling window explains why your score remains unchanged. CrUX serves as the official record for your Core Web Vitals, but it is not built for active development monitoring. To verify if a deployment worked on the same day, you need a RUM tool to track live sessions.
Advantages:
- Automatically collects experiences of real users who visit your site on Chrome
- Matters for SEO via the Page Experience ranking signal
- Provides a broad overview instead of detailed insights which a RUM can
- Measures speed (TTFB, FCP, LCP), visual stability (CLS) and interactivity (INP)
- The official dataset for the Core Web Vitals program
Disadvantages:
- Data collected over the last 28 days is always delayed
- Limited to Chrome browser data only, excluding all other browsers
- Cannot group per template, only origin or page URL
- Only includes data from opted-in users and sufficiently popular, publicly discoverable sites
- Includes data from iframes and cross-origin resources
Although CrUX-based tools are a good starting point for investigating Web Vitals performance, we strongly recommend supplementing it with your own RUM.
If you want to know whether a fix worked today, check your RUM tool. If you want to know whether Google has registered it, check CrUX in three weeks.
RUMvision vs. Google Lighthouse
The biggest difference between Google Lighthouse and RUMvision is the fact that Lighthouse operates with lab data and RUMvision with field data. Therefore it measures two completely different things. We wrote a blog about "When to not use Google Lighthouse" to get a better understanding of its use-cases.
Often Lighthouse's data comes from a device with slower internet than your own users, as such, it might not accurately represent the experience of your users.
Advantages:
- Always free and available (even in staging environments)
- Gives diagnostics and tips
- Gives a clear overall score which everyone is crazy about
Disadvantages:
- Not based on real users experiences
- Unlike a real user, Lighthouse does not interact with your site, therefore it cannot measure INP at all
- Not used for SEO ranking
- Easily and often faked because people think it is a business metric, which it should not be
- Can only identify one LCP candidate and does not take TTFB into account
- CLS is not tracked beyond first load
- Only simulates a first load and does not factor TTFB into the overall score
During a Lighthouse test the "user" does not interact with the page, this could introduce blind spots for metrics like the Cumulative Layout Shift (CLS) and Interaction to Next Paint (INP) that can happen at any given time. It only measures the first 5 seconds of a page load. The closest Lighthouse has to offer for the INP is Total Blocking Time, but the results will also differ from what real users experience.
If you want an idea of which script was actually harming during user interaction, with a button for example, the easiest way is to install RUM.
Apple.com score on Google Lighthouse
RUMvision vs. GTmetrix
Even though GTmetrix technically uses lab data they also try to simulate a real user's experience and behavior on a site. You can change these "user's" variables like location, connection speed, etc. By doing this GTmetrix gives you the option to select variables of what you think represents your own audience. Keep in mind that this could introduce blind spots, as you will not know which type of visitors experience bad Core Web Vitals on your own website.
The GTmetrix score can also differ from other lab data tools like Google Lighthouse and WebPageTest, because all of them have a different methodology.
With RUMvision you can see which type of visitors you have on your website and how this correlates with web performance metrics. Not every user will be on a fast or slow device, but you have in-betweens as well. Therefore it is useful to capture visitors by the fact that they entered your website via an ad, are they first time visitors, what their internet speed is, and how fast their device is.
RUM data via the web-vitals.js vs RUMvision
You can also measure real user monitoring and field data yourself using the web-vitals.js library.
Advantages:
- Real time
- Also based on real visitors UX, even including other browsers.
- Maintained and constantly improved by the Google Chrome team.
- Data reliability only increases in case of real visitors.
- Open-source and GDPR friendly
- Can save a lot of time and money debugging performance issues.
Disadvantages:
- Not all web vitals are supported in all browsers (CLS for instance is both missing in both Safari and Firefox, SPA soft navigation tracking is only supported by Chromium)
- Will provide a lot of technical details, such as exact DOM-node
- Is not a substitute of Lighthouses' / lab data diagnostics
- Sending, saving, updating and visualizing can be tedious and costly
- You miss dimensions (country, browser, device type, viewport), making it hard to reproduce and fix issues
Commercial RUM (RUMvision)
A commercial RUM shines in effortless implementation and rich and detailed visualization, which is why Google recommends using us.
RUMvision is recommended by Google as a RUM solution
Advantages of using RUMvision
- Monitor and compare Core Web Vitals +40 different metrics by template in real-time + add custom timing.
- Set annotations on new releases, see trends and receive alerts
- Developed and hosted in the EU, no PII + GDPR proof
- Analyze direct and 3rd-party JavaScript execution effects on INP
- Software with a service, we provide free quick wins and on-boarding, guiding you where to improve web performance
- Based on the web-vitals.js, but without work, just install a small JS snippet and you're done
Disadvantages:
- You pay for a subscription (prices start at 150 euro per month)
- Limited access to data within iframes due to security restrictions
- Development is determined by our own roadmap
From CrUX to RUM: Your next step
If you started your journey today by checking your domain in our free CrUX tool, you now know exactly what Google sees. But knowing your score is only half the battle.
To actually improve those metrics, you need to look under the hood of your real user sessions. You need to know which specific images are delaying your LCP, which third-party scripts are causing layout shifts, and how performance fluctuates between your fast mobile users and your slow desktop users.
You do not have to wait 28 days to see if your site is getting faster. Start collecting real-time field data today to see exactly what your users are experiencing right now.
