Last week I talked about how we chose Wraith as our visual regression testing tool.
After lots of discovery and preparation, we were now ready to get our first real tests up and running.
Example of a failed test in Wraith. Differences are highlighted in blue – in this case, I've modified the summary slightly.
In this post, I'll go through how we run our tests locally and as part of our continuous integration process. But go make a cup of tea first, because this will be a long one...
We've recently been working on getting visual regression testing up and running for bath.ac.uk.
Back in October, Liam blogged about our earlier research into visual regression testing. But if you need a refresher...
Visual regression testing is a way of automatically checking webpages to catch any unwanted changes, or 'regressions', in the design.
These unwanted changes are a risk we run every time we modify the code for the design of the site. Sometimes a single tweak in one place can have unexpected consequences for another aspect of the design.
We could just eyeball lots of different pages to see if anything looks wrong, but this takes a lot of time, and there's always a chance of missing some tiny difference.
It would be much better to write some automated tests which take screenshots of the pages before and after we've modified our code, and then highlight any changes.
Differences are highlighted in blue. Oops, that should not have moved.
Basically, it'll save us loads of time and keep bath.ac.uk looking great!
We were excited to crack on, but before we could get our tests up and running, we had some decisions to make.
There are a lot of different options out there for visual regression testing. We'd previously followed a tutorial using WebdriverCSS, but struggled to make the tests run consistently and generally found it a frustrating experience.
We decided to investigate alternatives. Our discovery process included:
Some of the tools we looked at were:
After doing the research and discussing the options with our designers and developers, we decided to go with Wraith.
We ain't afraid of this ghost
Built by BBC News developers, Wraith has also been used by the Guardian and the Government Digital Service.
It's written in Ruby, our dev team's language of choice. The config and tests are written in YAML, which we felt was easier to work with than the alternatives (usually JSON or JavaScript). When we tested it out, it took less than an hour to install everything and write and run our first tests.
It also offered two options for running the tests – you can compare new screenshots to historical ones, or compare the same page across two different environments (for example, staging and production).
We'd chosen a tool, but we still needed to decide what to use it on.
Like many visual regression testing tools, Wraith allows you to test entire pages or individual elements.
Testing individual elements makes finding differences much more precise – otherwise one change high on the page can have a knock-on effect on everything below it. Having a comprehensive pattern library can make it even easier to test individual elements.
All of this sounded great, but our pattern library isn't in a testable format yet, and writing tests for individual elements is time-consuming.
We wanted to get up and running as quickly as possible, so we decided to test full-page screenshots of real pages in each content type. This will provide a solid level of coverage, and of course we can always keep improving our tests and making them more comprehensive.
We also had to decide what screen resolutions to test at. We checked our Google Analytics stats for 2016 and found the top 3 resolutions across desktop, mobile and tablet.
One size (1280 x 800) appeared in both desktop and tablet, so this left us with 8 sizes to test against – ranging from 1920 x 1080 all the way down to 320 x 568.
We now had a plan for writing and running our first real visual regression tests.
In my next post, I'll talk about the process of getting Wraith set up and fitting the tests into our workflow.
Ahoy there! Me name is Tegan and I be Digital’s fresh meat. I joined on the first o' December last year. I’m not long graduated from university, with twenty months in industry, tamin’ the wild seas of the web.
I’m a design buccaneer, a user experience swashbuckler. I conduct hearty user research, rescuin’ marooned users from the shores of confusion, and I make user interfaces delightfully easy to use. I also have a fair working knowledge of code which helps me parlay with devs.
Batten down the hatches an’ I’ll tell ye a little tale ‘bout me first big ship.
</pirate>
When we release a new feature or change, we call it ‘shipping’.
This ship had a mission: to grab your attention, inspire you to keep reading and build upon the University of Bath’s visual language with a new visual cue.
This element will be used by Collections, thematic pages with handpicked articles. But it may expand to other aggregate content types, like departmental landing pages.
We’ve called this new element a Hero – it’s well-built and displays outstanding achievements and noble qualities! I kid, it’s actually a technical term for a prominent banner on a page.
Hero image is a term used in web design for a specific type of web banner, prominently placed on a web page, generally in the front and centre.
To start off the design process, we collected data from current users to see what their behaviour is currently.
Using the scroll heatmap, we could see where people linger on the page.
This heat distribution demonstrates that people don’t spend much time above the fold (the top bit of the webpage that is visible without scrolling down). Users quickly scroll below the fold to read the actual content. This is the hottest spot on the entire page.
Next, we checked out how other people do it. What are the current standards, what does and doesn’t work? This process is called peer review and it is a great opportunity to draw inspiration from what other people have done.
Stop. Sharpie time!
I squirrelled away in a meeting room with the collection of Sharpies and A3 paper to punch out weird and wonderful ideas and directions.
I developed a handful of these ideas further, then mocked them up as high fidelity prototypes and presented them to the product owner for feedback.
Armed with feedback, I iterated on my designs with more sketching and more mock-ups to bring back some new and improved ideas to the table.
I collaborated with Sharpie champion and user experience privateer, Dan, to create a new piece of visual language for featured images. This hinted at the existing call-to-action arrow shape with a bit more visual embellishment.
14 artboards later, we were ready to demonstrate the design on a multitude of device sizes and its limitations. These were shown to the Director of Marketing & Communications for final approval.
Sail ho! The design was approved!
To breathe life into this new element, we needed to wire it up to the Content Publisher. This required input from multiple disciplines: designers, developers and content producers.
My role was to create the front-end by gluing HTML and Sass together, while Iris developed the working bits to add a new hero section in the editor, and content seadogs composed some explanatory microcopy.
Collaboratively working with Iris was a breeze, and each piece of work slotted together like a jigsaw puzzle. The build took 2 days, with a further day of squashing bugs and browser compatibility testing. 36 git commits later, we were ready for review.
As this was my first time working with both the Foundation framework and the university's visual language, Origins, I think I did good! Some code structure optimisation and minor (but significant) design changes came to light in review, but all the required amendments were quick and soon got the green light.
Pull requests to production put the build into motion. It was time to weigh anchor and set sail!
The deployment dragon roared out across the sea (department) as the ship... shipped.
This feature was a huge blocker for launching the Research collection, so having it ship was an achievement for the whole crew. You can now see the Research Collection live, or read Hanna's post about it from a content perspective.
This is, admittedly, not my first ever ship here at Digital. So far I have set sail to several little ships, or... dinghies, but this feature was my first big one. Drawing upon my own skills and those of the people around me, we have created a cool new element to drive forward a new chapter in the University of Bath’s visual language development.
I raise me grog to the crew and all future ships!
Yo ho, yo ho, a pirate’s life fer me...
It’s been almost 3 years since I joined the University of Bath digital team and, despite committing wholly to the agile workflow, I’m still struggling to reconcile certain aspects of our working practice with the freelance life I left behind.
As a designer with almost 20 years of experience and weened on a thoroughly waterfall design process the thought of handing over ‘unfinished’ work to a stakeholder is still an uncomfortable one.
In ‘The Good Old Days’ done meant done.
As an interactive designer I’d often be tasked with producing dozens of pixel-perfect page designs illustrating every conceivable content type, state and variant. These layouts were honed, polished, considered works-of-art (at least in my mind).
And these designs were considered final. The end of the design process. I was done. These static snapshots of a site were then presented to project stakeholders as a fait accompli.
“Do you like the design? This is what your site will look like”
“Yes. It looks great.”
“Wonderful. Summon the developers - we have a site to build!”
In the Digital team, we use agile methodologies when designing and developing. In basic terms, this means that we are constantly iterating, improving and updating small, discreet elements of our designs based on feedback.
The upshot of this is that the first version, the thing you launch, the live site, is not perfect. In fact it never will be.
The MVP (Minimum Viable Product) is a base-level, functional version that lets the user do what they need to complete a task or locate a piece of information. For a designer, used to polishing a design within an inch of its life before letting anyone see it, the MVP is a rude awakening.
Things aren't perfect, they aren't the best they could be. They most certainly aren't finished. However they do meet the users needs for a functional, viable experience.
The lack of control still makes me feel nervous but designing for a specific user need has changed the way I think about my work.
It can sometimes seem like an agile project has no plan, no overall vision for the end result.
From a design perspective, this can be frustrating when working on the conceptual elements of a design. Trying to get a grip on what *everything* looks like and how it holds together can be tricky when working on a single aspect of an interface component or tiny piece of functionality in seeming isolation.
The understanding of when an MVP can be described as viable is a skill in itself. The PO (Product owner) holds the product vision and is the arbiter of 'ready' - they say if a component, a template or a feature is in a position to ship.
And shipping happens as often as possible. One of the basic tenets of agile is that we look to get things in front of real users as quickly as possible. Any feedback the users provide is then used to refine or create further user stories to iterate on the previous work and make it better.
I may not have my pixel perfection but I do have real, genuine data from those on the coal-face using the things I have designed.
Putting things in front of people is still scary after all this time. No-one likes criticism. We certainly don’t like watching anything we have worked hard on fail.
The flip-side of this is that user-testing is a liberating, eye-opening experience for any designer. Watching a user struggle to find your meticulously crafted call-to-action or swiftly complete a complex multi-stage task brings you closer to your users.
I love this new-found connection with the end users of the products and services I design.
… is an awkward hug with good intentions.
Learning to embrace agile has certainly been a challenge. Three years in and my user stories could always be better, my fear of bad feedback has only slightly diminished and my frustrations at shipping something that (I think) looks like a dog has chewed it a bit have not abated.
However, the openness of communication and the interaction with the people who use the things I make on a day-to-day basis have meant that I believe my design skills have improved immeasurably.
Maybe it is possible to teach an old dog new tricks.
Last month, I made a trip to the Library to delve into the archives of the University. Together with colleagues in Imaging, Design and Printing Services (IDPS), the plan was to get a crash course on brand design.
The head archivist, Lizzie, had sourced boxes full of printed materials for us, ranging from prospectuses to annual reports, promotional posters and pamphlets.
As designers we were first drawn towards the prospectuses. Published annually, it provides a fascinating insight into the prominent design trends in the year of publication. Our 90s 'rave' period is probably one to forget.
As is often the case, our most interesting discovery came from a rather unexpected source.
In 1965 an unassuming document outlining plans for the new University in Bath was published. With a brown and cream two-tone cover and type set entirely in Helvetica (known at this time as Neue Haas Grotesk) Development Plan Report Number 1 wears its 1960s origins proudly on its sleeve.
The report itself is a beautifully typeset outline of the plans for the proposed University of Bath. It’s stuffed full of abstract, organic diagrams and line drawings that are almost cellular in their approach. Throughout the report living, studying and social spaces are referred to as organic entities, flexing and changing to suit the needs of their occupants.
The first real surprise adorned the back cover - A large black and white photograph of the Sulis head found in the local Roman baths.
Bas-relief from the pediment of the Temple of Sul Minerva in Bath
The creative decision to position this particular image here has had long-lasting ramifications for our design direction. The Sulis head has been inextricably linked with the University from its conception.
Which is why it was such a delight to discover a small hand-rendered version of the icon on the inside front leaf. This subtle design element cements the relationship between the nascent University and the Sulis head icon.
Sulis head icon on the inside cover
Having discovered the impact the report had made on the University's brand identity we dug deeper, looking for yet more insight. In the first section - Aims and Principles - we found 'The first diagram of the university' (the actual title of the image).
The first diagram of the University
This incredible diagram illustrates a number key design tenets repeated throughout the report - namely a campus that grows outwards from a central hub (the spine/nucleus), close integration of residential, academic and social spaces and restrained (but not constrained) campus structure.
The beauty of this diagram is how well it effectively communicates relationships between traditionally disparate spaces. The campus is visually described in anatomical terms - It feels like a living entity, exactly as intended.
Development Plan Report Number 1 eveals that the University of Bath was conceived from the very beginning as something new, something different, an institution to challenge.
The plan discusses possible residential and social patterns at length. Two of these patterns are based on the structure of existing academic insitutions - A traditional 'Redbrick' college and the 'Oxbridge' model. A third proposed pattern is considered new and unique to Bath.
Three patterns for University structure
In the Redbrick pattern there is little relationship between the academic activities of the departments and the social activities of the students. These elements remain isolated from each other.
With Oxbridge the pattern consists of a loose ‘community’ of independent colleges and scattered teaching departments. Colleges lose some of their academic significance to the departments and students live either within the college or in separate lodgings.
In the Bath pattern both study and socialising are closely linked in a concentrated urban setting. The university is conceived as a single varied community. There is no attempt made to separate or isolate the different facets of student life.
At the core of the proposed new campus is the parade. Designed to flex and grow as requirements change, the parade is never more than (about) 2000ft long - roughly 8 minutes walk from end-to-end at a leisurely pace.
Once again the thinking behind this concept it outlined in a series of beautiful abstracted images. Lacking in actual detail these illustrations still manage to convey a sense of movement and space between the edifices that make up the central area of the campus.
That the last illustration (the selected option) is still recognisable as the parade today is incredible.
Flow patterns through the parade at the heart of the University
It was clear from just the short time we had to spend with the Design Plan Report Number 1 that things hadn’t changed as much as we might have originally thought. The founding principles of the University still hold true today including the Sulis head representing the University at a core level.
The beautifully executed line drawings summon up an image of a sophisticated, unified campus blending all aspects of student life into a seamless experience.
We came away inspired, motivated and with renewed optimism that the University brand was not far off alignment.
Our new site consists of 15 different layout templates. Each one of these is further broken down into numerous different design patterns for consistently displaying content. The rules that govern the presentation of these patterns (or elements, if you are familiar with atomic design) are generated from a combination of the Zurb Foundation framework and our own 'Origins' framework - all in all over 2000 rules spanning almost 6500 lines of css.
With this level of complexity it is extremely hard work to track the effect of any changes, almost certainly there is an unexpected knock-on effect of changing, re-specifying, or removing a rule.
Up until now, we have relied on our in-depth knowledge of the site to know where we expect changes to appear, and we use Browserstack to quickly check a representative sample set of our layout templates.
However, this requires a fair whack of time to run, and also needs a person to sit and look at each snapshot that's generated. And they need to know what to look for.
None of that is optimal, we needed a way to automate this process. Enter visual regression testing.
We followed this online guide by Kevin Lamping for setting up a prototype for a visual regression testing framework for Origins and the site templates.
Our prototype is in a repository here: https://github.bath.ac.uk/digital/visual-regression-testing/ (you will need a University of Bath login to view this). It contains a README with instructions on how to get it up and running.
Essentially, what it does is use some clever existing technology (WebDriverIO, WebDriverCSS, graphicsMagick, Selenium standalone and Node) to make a brower load a specific page, take a screenshot of a defined element, and then compare it to a baseline image. If there is any visual changes, it will then create a 'diff' file showing the change - and also alert us by throwing an error.
First we create a baseline image.
Then, when a change happens, another image is generated and compared against the first.
If there are any visual changes, a third file is generated showing these changes.
Currently we are manually running these tests, but ultimately we will integrate into our continuous delivery framework so that the test run automatically whenever a new build of the css is pushed to our staging server.
Pretty neat.
Based on our current Digital Delivery Principles, I recently sat down and drafted a set of Design Principles. These were shared at one of our Show and Tells as a presentation, but I thought I'd reproduce it here on our blog for a wider audience.
Every design decision we make should solve an actual problem that a site visitor has, or facilitate a real user need. Design should serve the content it presents, design in the absence of content is not design, it’s decoration. We always keep in mind that the end user of a page can be very different to us, with different needs, expectations, and abilities, and ensure that our design does not exclude them from getting the answers they need.
Our approach is inclusive: we're building a site that will provide the optimal experience for a visitor whoever they are and whatever device they are using. To achieve this we conduct user research to better understand what people expect. We test changes, features and assumptions with users to get insight and feedback that ensures we are delivering on that expectation. It is as important to remove failing features as it is to add new features into our design, and the ability to know what works and what doesn't only comes from data.
We maintain a pattern library to ensure the design of elements is consistent, and this ensures that our interactions speak to users with a single voice, building trust.
We strive to ensure that our design affords our visitor with an experience that feels fast – they should be able to find what they need on the page without a delay. Images are optimised for the device they are viewed on, visual enhancements are loaded progressively, and do not disturb the flow of content on the page.
We minimise pain-points as much as possible with an awareness of good practice, to provide an effortless experience.
The success of the University of Bath is based on calculated risk-taking, knowing when to break from convention, and when to reprioritise your approach to better fulfil the requirements of the people who depend on you. Instead of simply matching what other institutions offer, we challenge them with innovative approaches and ideas that better serve users’ needs.
It is pointless to sink 2 months into crafting the most beautiful interface if it does not allow the visitor to complete their task, or does not work on a mobile device. We release a design feature as soon as possible, and then iterate on that delivery to improve it.
Doing anything with more regularity will provide performance benefits and provide insights into how to do it better, and over the last couple years we've really ramped up the frequency of our user testing.
Sooo, here are five lessons that I want to share from our latest round of testing (for our online course publisher). They really helped me to improve our process.
We used to have our own 'testing lab' (a MacBook Pro running Silverback hooked up to PC peripherals), but we quickly found that people got confused by the unfamiliar interface.
Where possible we now get users to test on their own machines at their own desks to remove this barrier. This has the benefit of making our volunteers feel at ease by being in familiar surroundings. In other situations like guerilla testing, we've found that a tablet works really well, due to being portable and not requiring any accessory beyond a finger.
It's really hard to ask the questions and capture the answers effectively without help, so bring along a colleague. This also helps when approaching people cold because you will be more relaxed and confident, and from their perspective two random strangers approaching you is less unsettling than one.
Generally you get fewer in-depth responses when testing your product with a group of people, the trade-off being the greater breadth of replies. However, group discussion of individual responses leads to additional insight that you will not get when talking with one person. Make sure you allow time your testing script to accommodate this off-piste discussion.
It's better to test a few features thoroughly, than rush through a whole raft of different aspects. Also, it's really important not to run too many sessions at one time; it’s tiring and you’ll miss insight through fatigue and response bias (you re-interpret what someone says because you've heard the same response 8 times already that morning).
We found 4 large-scale (~45 minute) sessions was enough for one day - and around 9 smaller scale (~10 minute) sessions.
You're proud of what you've done, but you want to make it better. Don't be defensive of any issues, instead thank the user for finding the problem - they've done you a big favour.
Always remember that people are giving up their time for you, for no reward - so make it an enjoyable experience!
After a busy year, the onset of Christmas always provides a brief moment to step back and reflect upon everything that has happened over the last 12 months.
2015 has been our year for honing workflows, trying out new tools and technologies, and shipping, shipping, shipping.
This intensive regimen has left us shouting "This is great!" almost as much as "Why can't you just work, dammit!" Here are the top five things the Digital team design function would really like to find under our Xmas tree to make us feel all happy and fuzzy.
42 un-squashed commits in that PR? You have fun with that…
My chosen Git tool - Tower - lacks the functionality to make interactive rebases. This means I end up having to drop into the command line to squash commits for my PRs and that’s just asking for trouble.
Please Santa, can we have a well-designed, feature-rich Git GUI client with interactive rebasing for Xmas?
Vagrant has been a real boon allowing us to run up an exact copy of our publisher and editor apps on our local machines. However it can be a bit of a nightmare to manage if you don’t lean towards the technical side of things.
Please Santa, can we have a better way of managing/configuring Vagrant boxes that doesn’t mean I have to know what my $path is?
As a design team we’ve really gone hell for leather with regards to our implementation of web fonts this year. Although we’re very proud of what we’ve achieved, it has been hard work tackling cross-browser rendering issues, OpenType hassles, caching and pre-loading, font verification and much, much more.
Please Santa, can we have a standardised web font format that works perfectly on EVERY browser and platform?
We’ve pushed almost all of our preliminary design work through Bohemian Coding’s Sketch over the last 12 months. I've been working with Sketch since the heady days of version 1.0 but the software was completely new to Liam. Although he could see the benefits of the stripped-back interface and laser focus on interface design, he was constantly infuriated by Sketch’s little ‘eccentricities’ and general bugginess.
Please Santa, can we have a version of Sketch that’s stable, reliable and fixes that awful line-height bug once and for all?
Adobe’s last great hope? Maybe that’s a bit dramatic, but Project Comet is looking like a genuinely interesting contender in the race to grab the interface design crown.
Please Santa, Comet looks better than anything Adobe has made in ages. Can you get them to speed it up a bit? (Or at least get us on the Beta.)
Have a very merry Christmas and a happy new year! May all your workflow wishes come true!
It's almost 10 years ago now that Oliver Reichenstein published his thoughts on the fundamental influence of typography on web design.
Web Design is 95% TypographyOliver Reichenstein
At the time the article provoked a lively discussion amongst the design community. Today you're unlikely to find many voices disagreeing with Reichenstein – unless it is to argue for an increase in the quoted percentage by a few points.
Designing a typographic system to underpin the large-scale redesign of the University's website is a complex and involving task. The seeds of our thinking regarding a full typographic review have been germinating since early-2014 when Liam presented his thinking around the haecceity of Bath.
The development of the alphas provided an excellent opportunity to implement a vibrant new typographic approach for the University of Bath.
The first step was to outline the minimum criteria for our new typefaces. These criteria fit broadly into 3 categories:
A full, flexible character set
It was clear from previous flawed implementations of open source/free typefaces that a full and varied character set was a must. When assessing typefaces we looked for diacritics, non-Latin characters, beautiful ligatures, alternative characters and more.
Reliable, performant delivery method
Typefaces with multiple weights and variants can clock in anywhere up to 1MB in size for the full set. We needed configuration options to reduce loading time, the option to self-host on local servers and other methods of cutting down were all explored.
Rendering
A well-designed, crafted typeface should render optimally whatever device it is displayed on. Proper hinting and optimisation for screen display is vital.
Legibility and clarity
Our chosen typefaces must work at varying sizes and lengths of content. Clarity and legibility of text is paramount.
Personality
How does the typeface 'feel' when set on a page? Does the typeface genuinely reflect the University of Bath and the persona we have in mind?
Meaning
It's too easy to create a false relationship where none exists - we wanted typefaces with genuine meaning and subtext that enhanced the user experience.
With our criteria in place we started looking at typefaces. Many, many typefaces …
Over a handful of sprints we narrowed our typeface selection to a shortlist from foundries such as Dalton-Maag, Jeremy Tankard, Hoefler & Frere-Jones and other typographic luminaries.
To meet our technical criteria we elected to use typefaces in the Opentype format delivered as web fonts via fonts.com. Opentype gives us the flexibility to add or remove typographic features as and when we need them - key to reducing the initial load time for multiple fonts. The fonts.com web font delivery service is fully featured, robust and offers a variety of font hosting options.
Early on it was decided that we should work towards selecting typefaces from a single foundry. Selecting our typefaces from one source meant that we could be sure that each face would share a commonality in design-thinking, style and concept. These values help to ensure different typefaces work together harmoniously when paired on the page.
To this end, we have selected two typefaces from a well-respected, long-standing UK type house now based in the Netherlands, The Foundry.
One of The Foundry's principle designers David Quay has a history with Bath having been the lead on the team that developed the Bath city typeface used for signage, way-finding and infographics. The Foundry are well regarded for their work in creating legible, elegant typefaces that have a quintessentially English feel but global appeal.
Foundry Origin
A beautiful, modern slab/Egyptian serif for headlines
Foundry Origin - A quiet design with a big presence.
Foundry Sterling
A clean, legible sans serif for body copy
Foundry Sterling - A modern sans with a quintessentially English flavour
If you're quick, you can see the typefaces being used on alpha.bath.ac.uk and alpha.bath.ac.uk/about before the Alpha exercise comes to a close at the end of this week.