Accessibility Testing with Google Lighthouse

Accessibility reports are crucial when testing web pages because they help to ensure that a site is usable for people of all abilities. That is why we introduced Google Lighthouse to Neodymium, an open-source tool to improve the quality of web pages.

Install Lighthouse CLI

First of all we recommend installing a package manager like npm, which we are also going to use in order to install Lighthouse CLI. After you went through the installation process of npm, open a terminal and enter the command below.

npm install -g lighthouse

To make sure the Lighthouse installation was successful, you can run the following command.

lighthouse --version

Use Lighthouse Inside Neodymium

With the objective of creating Lighthouse reports inside Neodymium we implemented the class LighthouseUtils, containing the function createLightHouseReport(String reportName). By calling this function, a Lighthouse report of the current web page is generated and automatically added to the Allure report with the name specified in the reportName parameter. Keep in mind that creating a Lighthouse report only works while using Chrome or Chromium-based browsers.

The method call can be added at every point of your test and will create a report of the currently opened page. This also works for pages, which rely on session data like a login state or products that should have been added to a shopping cart. Keep in mind that it may not work on certain pages that, if refreshed, load a different page, like checkout pages for example.

Here is an example how to add createLightHouseReport() inside a test case:

@NeodymiumTest
public void testLoginAsRegisteredUser() throws Exception
{
    // go to homepage
    var homePage = OpenHomePageFlow.flow();
    LighthouseUtils.createLightHouseReport("Homepage");

    // go to register page
    var registerPage = homePage.header.userMenu.openRegisterPage();

    // send register form
    var loginPage = registerPage.sendRegisterForm(registeredOrderTestData.getUser());

    // send login form
    var accountOverviewPage = loginPage.sendLoginForm(registeredOrderTestData.getUser());
    accountOverviewPage.validateSuccessfulLogin(registeredOrderTestData.getUser().getFirstName());

    LighthouseUtils.createLightHouseReport("Account Overview Page");
}

Please note that this example uses page object model classes which are not shipped with neodymium.

Limitations

Please note, that Lighthouse does not work with modals, fly-ins, hover etc. The site will be newly loaded for the report, so every manually opened modal will be closed after a Lighthouse report. This needs to be considered when generating reports and might require changes to the test flow (e.g. if a modal is open which contains the next button the scripts wants to click).

Lighthouse Reports

The generated Lighthouse reports can be found in the target directory of the user’s repository and in the Allure report as an attachment, visualized in the following image.

Lighthouse report inside an Allure report.

A Lighthouse report consists of the following four categories.

1. Performance
2. Accessibility
3. Best Practices
4. Search Engine Optimisation (SEO)

Each of those categories is scored between 1 and 100, which reveals how well the web page performed in every category. Therefore, Google defines the following ranges.

• 0 to 49: Poor
• 50 to 89: Needs Improvement
• 90 to 100: Good

Lighthouse Report Validation

This section is about how to assert metrics from the Lighthouse report.

Lighthouse Category Score Validation

To enable validating the scores of all four categories, we implemented the following score thresholds in the neodymium.properties file.

• neodymium.lighthouse.assert.thresholdScore.performance
• neodymium.lighthouse.assert.thresholdScore.accessibility
• neodymium.lighthouse.assert.thresholdScore.bestPractices
• neodymium.lighthouse.assert.thresholdScore.seo

All of those configuration properties are set to 0.5 per default, which sets all category score threshold to 50. That means each and every category needs to match or exceed a score of 50 or otherwise the test will fail. All the score thresholds can be changed depending on the user’s wishes.

Please be aware that the Lighthouse performance score is not the most stable value and is affected by many factors like network load on th the test machine. This can lead to random outliers in the measurement making your tests flaky. A check against the 75th percentile is recommended to give the test some stability (see for example here).

Lighthouse Audit Validation

We also implemented the property neodymium.lighthouse.assert.audits in the neodymium.properties file. This property makes it possible to validate Lighthouse audits. In order to do that the user has to specify the id of all audits that should be validated as the property itself. For example: neodymium.lighthouse.assert.audits = aria-roles aria-text validates that no error occurs in the Lighthouse audits aria-roles and aria-text. All existing audit ID’s and their corresponding titles are visualized in the table below.