Four simple automated accessibility testing tools
Good automated tools are a cheap way to help create and maintain a baseline of accessibility (a11y) in a product, to then be expanded on with manual testing.
We will look at manual testing strategies in another post, and here introduce 4 automated tools that can be used at different stages of the development process.
We will be looking at:
Why use automated tools?
Rather than re-write an already great explanation, let me directly quote from a UK Government Digital Services post:
Automated tools can be a useful and cheap way of helping you make a service more accessible. They are quick to run and provide immediate feedback. They can be run across lots of pages. Some can be integrated into the build process, so they can identify issues almost as soon as they are created.
But while it can certainly be helpful to run an automated testing tool on a service, it’s important that teams don’t rely on them too heavily. No tool will be able to pick up every accessibility barrier on a website. So just because a tool hasn’t picked up any accessibility issues on a website, doesn’t mean those issues don’t exist.
A good analogy is to think of a testing tool as like using a spellchecker. It can certainly help you pick up issues, but it should never be used in isolation. To be most useful, automated tools should be combined with manual inspection and user research.
The audit they published in February 2017 (please comment below if you know of a more recent one) showed only around ~30% of issues were caught by automated tools.
Due to their ease of setup and use this still makes it worth the small initial investment in adopting a tool. Of importance is that it should always be communicated to stakeholders that automated tools should not be relied on in isolation — to avoid the risk of those not as close to the product’s development making incorrect assumptions.
Google Lighthouse Audits
Lighthouse is an open-source, automated tool that can be run against any web page, whether in production or development.
It contains an accessibility audit option, and can be ran from Chrome DevTools, from the command line, or as a Node module.
You give Lighthouse a URL to audit, it runs a series of audits against the page, and then it generates a report on how well the page did.
This is useful for analysing a whole page and in comparing many pages against each other. It however is, currently, more of a high level tool that acts as a great add on to the existing Lighthouse audits.
Jest axe
jest-axe is an npm package that wraps axe-core for use in Jest tests.
It takes HTML input and runs axe tests against it alongside other Jest tests, outputting any failures to the console and failing a build in the same way as a standard test. This helps prevent regressions as it becomes part of the Continuous Integration pipeline rather than a one off additional check.
It allows all axe-core options to be specified from within the axe() function (for setting the accessibility standard or specific rules for example). It can be ran at a component or page level as appropriate.
import React from 'react';
import ReactDOMServer from 'react-dom/server';
import { axe } from 'jest-axe';
import 'jest-axe/extend-expect';it('has no programmatically detectable a11y issues', async () => { // pass anything that outputs html to axe
const html = ReactDOMServer.renderToString(
<MyComponent />
); const results = await axe(html);
expect(results).toHaveNoViolations();});
Pa11y CI
pa11y-ci uses HTML_CodeSniffer for detecting a11y issues. It is run against a full page and allows ‘actions’ to be used within a test to interact with the page under test. It also allows multiple pages to be specified within one configuration file.
I wrote about Pa11y CI in with more detail and setup instructions in a previous post.
"urls": [
{
"url": "http://pa11y.org/",
"viewport": { "width": 320, "height": 480 },
"actions": [
"wait for element .site-brand to be visible",
"screen capture screenshots-output/mobile-main-view.png",
"click element .site-nav__item button",
"screen capture screenshots-output/mobile-expand-menu.png"
]
}
],Errors are then output to the console with a description as below.
This requires slightly more setup than jest-axe, and can only be ran at a full app level, but its use of ‘actions’ gives it great versatility.
Storybook addon-a11y
Storybook is an open source tool for developing UI components in isolation. Whether developing a component inside an application, or for publishing as an NPM package, it allows you to run and view your component in the browser during development. It uses ‘addons’ to give optional extra functionality. This may be to be able to alter prop values within a React component, or, in our case, to have real time accessibility feedback.
For v5 of Storybook and above adding it is as simple as adding the below to your .storybook/config.js:
import { configure, addDecorator } from '@storybook/react';
import { withA11y } from '@storybook/addon-a11y';addDecorator(withA11y)
In version 5.0.11 this gives two main tools.
Firstly a colour blindness simulator to give instant feedback on what your component looks like to those with various categories of colour blindness.
Secondly the ‘Accessibility’ tab along the bottom gives real time feedback of accessibility standards. This allows you to pick up issues very early, and so communicate them back to designers and product people early on. These checks only happen while Storybook is running, rather than being part of the CI pipeline.
Final thoughts
These are the four main tools I am personally experimenting with at the moment, and so far my experience has been positive.
For applications where I control the whole app I have found pa11y-ci to be great, though a little steeper learning curve.
When I am creating an npm-installable component then the Storybook addon and jest-axe are what I use, as Pa11y has no running app on a URL to work with.
In applications that pull in their own installable components that I do not have control over I also lean towards using jest-axe at a component level against my own components. This is as otherwise the external components may fail CI and require globally disabling rules for a build to pass, which may lead to ones that I have control over being missed.
Lighthouse I find useful for explaining the notion of accessibility checks to others, quickly able to demonstrate against any page an example of an accessibility report.
Finally I will quote the GDS post linked above to emphasise:
For the most effective accessibility testing, we advise teams to combine automated tool testing with manual checking, an accessibility audit and user testing.
I hope you found this a useful introduction, and please comment below with any tips relating to these, or other tools, if you have experience with them.
Thanks for reading! ⚡️
Other posts I’ve written include:
