Choosing a visual regression testing tool

Posted in: Design, Development

We've recently been working on getting visual regression testing up and running for

Back in October, Liam blogged about our earlier research into visual regression testing. But if you need a refresher...

Visual regression testing in a nutshell

Visual regression testing is a way of automatically checking webpages to catch any unwanted changes, or 'regressions', in the design.

These unwanted changes are a risk we run every time we modify the code for the design of the site. Sometimes a single tweak in one place can have unexpected consequences for another aspect of the design.

We could just eyeball lots of different pages to see if anything looks wrong, but this takes a lot of time, and there's always a chance of missing some tiny difference.

It would be much better to write some automated tests which take screenshots of the pages before and after we've modified our code, and then highlight any changes.

Changes are highlighted in blue. Oops, that shouldn't have moved.
Differences are highlighted in blue. Oops, that should not have moved.

Basically, it'll save us loads of time and keep looking great!

We were excited to crack on, but before we could get our tests up and running, we had some decisions to make.

Choosing a tool

There are a lot of different options out there for visual regression testing. We'd previously followed a tutorial using WebdriverCSS, but struggled to make the tests run consistently and generally found it a frustrating experience.

We decided to investigate alternatives. Our discovery process included:

  • reading any guides and documentation
  • checking the release history to see how well maintained it was
  • getting it running locally if possible
  • writing and running some basic tests

Some of the tools we looked at were:

And the winner is...

After doing the research and discussing the options with our designers and developers, we decided to go with Wraith.

We ain't afraid of this ghost
We ain't afraid of this ghost

Built by BBC News developers, Wraith has also been used by the Guardian and the Government Digital Service.

It's written in Ruby, our dev team's language of choice. The config and tests are written in YAML, which we felt was easier to work with than the alternatives (usually JSON or JavaScript). When we tested it out, it took less than an hour to install everything and write and run our first tests.

It also offered two options for running the tests – you can compare new screenshots to historical ones, or compare the same page across two different environments (for example, staging and production).

Deciding what to test

We'd chosen a tool, but we still needed to decide what to use it on.

Like many visual regression testing tools, Wraith allows you to test entire pages or individual elements.

Testing individual elements makes finding differences much more precise – otherwise one change high on the page can have a knock-on effect on everything below it. Having a comprehensive pattern library can make it even easier to test individual elements.

All of this sounded great, but our pattern library isn't in a testable format yet, and writing tests for individual elements is time-consuming.

We wanted to get up and running as quickly as possible, so we decided to test full-page screenshots of real pages in each content type. This will provide a solid level of coverage, and of course we can always keep improving our tests and making them more comprehensive.

We also had to decide what screen resolutions to test at. We checked our Google Analytics stats for 2016 and found the top 3 resolutions across desktop, mobile and tablet.

One size (1280 x 800) appeared in both desktop and tablet, so this left us with 8 sizes to test against – ranging from 1920 x 1080 all the way down to 320 x 568.

Next up: implementation

We now had a plan for writing and running our first real visual regression tests.

In my next post, I'll talk about the process of getting Wraith set up and fitting the tests into our workflow.

Posted in: Design, Development


  • (we won't publish this)

Write a response