Accessibility Testing with Google Lighthouse

Accessibility reports are crucial when testing web pages because they help to ensure that a site is usable for people of all abilities. That is why we introduced Google Lighthouse to Neodymium, an open-source tool to improve the quality of web pages.

Install Lighthouse CLI

First, it is recommended to install a package manager like npm, which will be used to install Lighthouse CLI. After installing npm, open a terminal and enter the command below.

npm install -g lighthouse

To make sure the Lighthouse installation was successful, you can run the following command.

lighthouse --version

Use Lighthouse Inside Neodymium

To create Lighthouse reports inside Neodymium, the class LighthouseUtils is used, containing the function createLightHouseReport(String reportName). By calling this function, a Lighthouse report of the current web page is generated and automatically added to the Allure report with the name specified in the reportName parameter. Note that creating a Lighthouse report only works while using Chrome or Chromium-based browsers.

The method call can be added at every point of your test and will create a report of the currently opened page. This also works for pages, which rely on session data like a login state or products that should have been added to a shopping cart. Keep in mind that it may not work on certain pages that, if refreshed, load a different page, like checkout pages for example.

Here is an example how to add createLightHouseReport() inside a test case:

@NeodymiumTest
public void testLoginAsRegisteredUser() throws Exception
{
    // go to homepage
    var homePage = OpenHomePageFlow.flow();
    LighthouseUtils.createLightHouseReport("Homepage");

    // go to register page
    var registerPage = homePage.header.userMenu.openRegisterPage();

    // send register form
    var loginPage = registerPage.sendRegisterForm(registeredOrderTestData.getUser());

    // send login form
    var accountOverviewPage = loginPage.sendLoginForm(registeredOrderTestData.getUser());
    accountOverviewPage.validateSuccessfulLogin(registeredOrderTestData.getUser().getFirstName());

    LighthouseUtils.createLightHouseReport("Account Overview Page");
}

Please note that this example uses page object model classes which are not shipped with neodymium.

Limitations

Please note, that Lighthouse does not work with modals, fly-ins, hover etc. The site will be newly loaded for the report, so every manually opened modal will be closed after a Lighthouse report. This needs to be considered when generating reports and might require changes to the test flow (e.g. if a modal is open which contains the next button the scripts wants to click).

Lighthouse Reports

The generated Lighthouse reports can be found in the target directory of the user’s repository and in the Allure report as an attachment, visualized in the following image.

Lighthouse report inside an Allure report.

A Lighthouse report consists of the following four categories.

1. Performance
2. Accessibility
3. Best Practices
4. Search Engine Optimisation (SEO)

Each of those categories is scored between 1 and 100, which reveals how well the web page performed in every category. Therefore, Google defines the following ranges.

• 0 to 49: Poor
• 50 to 89: Needs Improvement
• 90 to 100: Good

Lighthouse Report Validation

This section is about how to assert metrics from the Lighthouse report.

Lighthouse Category Score Validation

To enable validating the scores of all four categories, we implemented the following score thresholds in the neodymium.properties file.

• neodymium.lighthouse.assert.thresholdScore.performance
• neodymium.lighthouse.assert.thresholdScore.accessibility
• neodymium.lighthouse.assert.thresholdScore.bestPractices
• neodymium.lighthouse.assert.thresholdScore.seo

All of those configuration properties are set to 0.5 per default, which sets all category score threshold to 50. That means each and every category needs to match or exceed a score of 50 or otherwise the test will fail. All the score thresholds can be changed depending on the user’s wishes.

Please be aware that the Lighthouse performance score is not the most stable value and is affected by many factors like network load on th the test machine. This can lead to random outliers in the measurement making your tests flaky. A check against the 75th percentile is recommended to give the test some stability (see for example here).

Lighthouse Audit Validation

We also implemented the property neodymium.lighthouse.assert.audits in the neodymium.properties file. This property makes it possible to validate Lighthouse audits. In order to do that the user has to specify the id of all audits that should be validated as the property itself. For example: neodymium.lighthouse.assert.audits = aria-roles aria-text validates that no error occurs in the Lighthouse audits aria-roles and aria-text. All existing audit ID’s and their corresponding titles are visualized in the table below.

The following sections list all available audits in a Lighthouse report. The type column values are defined as the following:

1. Metric
   • What it is: These are the heavy hitters of performance monitoring (often called “Core Web Vitals”). Unlike a simple “Pass/Fail” check, a Metric measures a specific value (usually time in milliseconds or a layout shift score).
   • Data Provided: It provides a raw numericValue (e.g., 1200 ms) and maps it to a score (0–1) based on a log-normal curve.
   • Examples: first-contentful-paint (FCP), cumulative-layout-shift (CLS), total-blocking-time (TBT).
2. Audit (Standard/Binary)
   • What it is: These are automated checks that verify if a specific best practice is followed.
   • Data Provided: Usually a binary score. 1 = Pass or 0 = Fail.
   • Examples: doctype, is-on-https, image-alt.
3. Manual
   • What it is: These are items that Lighthouse cannot test automatically. It detects that the element exists (e.g., " You have a custom button") but cannot verify if it works correctly for a screen reader user.
   • Data Provided: The score is usually null or 0 in the raw JSON report because the computer cannot “pass” it for you.
   • Examples: visual-order-follows-dom, focusable-controls, custom-controls-labels.
4. Diagnostic
   • What it is: These items do not “pass” or “fail.” They are purely informational buckets of data used to help you debug why a Metric might be low.
   • Data Provided: They return a score of null (or sometimes 1 just to show they ran), but the real value is in the details object (lists of URLs, nodes, or request chains).
   • Examples: network-requests (List of every file downloaded), critical-request-chains (Tree of resources blocking the render), main-thread-tasks (Breakdown of CPU time).

Performance Audits

These audits measure metrics and suggest optimizations to improve page speed and user experience

Accessibility Audits

Lighthouse uses the axe-core library for most of these checks. These ensure your site is usable by people with disabilities.

Best Practices Audits

These audits check for general code health and modern web standards.

SEO Audits

Checks that ensure your page is discoverable and optimized for search engine crawlers.

PWA (Progressive Web App) Audits

Checks for PWA validation, including installability and offline capabilities.