Written by Sam Street (UX Designer) and Daniel Matongo (Back-end Developer).
What is the URL shortener?
The URL shortener is an in-house app that the Digital Development team built some time ago. The name sums its purpose up pretty neatly – we use it to generate short URLs much in the same way that similar apps like Tiny URL or Bitly do. The use cases for short URLs at the University of Bath are varied but the main one is for use in important publications such as posters, ad campaign assets, and prospectuses where marketers, understandably, don’t want to print a full URL as they can sometimes be quite long and look messy.
You might be asking yourself why we built our own app when other solutions like Bitly already exist. Well that’s fairly simple – having our own app gives us greater control over short URLs, in particular, managing access, and helps keep our costs down.
Why did it need to be rebuilt?
The main reason behind the rebuild was to address the lack of SSL certificates while at the same time, taking the time to make the app more efficient and performant – a predominantly back-end job. The original app was written in Ruby, which, while a great language in its own right, began to introduce problems as the application grew in size. We found ourselves unable to upgrade the application’s libraries without breaking it and making other improvements was too difficult due to the age of the codebase.
Starting at the end and working backwards
As we’ve said, the rebuild was largely a back-end focused job. This, together with the fact that the team has a lot of work on, led one of our back-end developers to tackle this project mostly on his own as ‘work-on-the-side’. What this resulted in was a robust solution from a technical perspective, but one that had no input from anyone else on the team.
User experience (UX)
The URL shortener hadn’t received any real UX attention since it was first built. This is down to the fact that it’s an in-house app that is only used by a handful of staff (i.e. internal users) and it’s rarely been a priority relative to other work. However, internal users are still users too and they deserve to use well-thought-out products so, we decided to conduct a usability audit on what had been built.
We decided to give the new app its own design language. One that was simple, but cohesive and based on principles used in other internal products to give it a familiar feel. We hoped this would help toward building trust in its users.
Beyond the look and feel, most of the UX attention went into ensuring the new app was easy to use. In practice, this meant three things:
- making sure common, and repetitive tasks were easy
- making sure the status of the system was always clear
- following established UX patterns
Working with the URL Shortener tends to involve lots of repetitive tasks such as batch deleting short URLs. We made sure that the new app provides functionality to make batch processing quicker and easier.
There are also lots of common tasks which were difficult to carry out in the old app. To pick on a specific example; one of the key things our Digital Supporter always checks when creating new short URLs, is if the requested short name is already in use, or if there is another short name for the same URL that could be used instead. To find this information out in the old app, they would have had to run a database query using another program. Now, in the new app, when a new short URL is created, the app will automatically check to see if either the short name, or the redirect URL is in use elsewhere and inform the user.
There are also many long established UX patterns – ways of presenting common information or workflows – that the old app didn’t use, for a variety of reasons. Things like no confirmation modals when deleting items (or even user accounts), not having the same behaviour in different pages of the app, etc. Now, most of these types of tasks use common UX patterns including things like:
- presenting search results without having to hit the return key
- always asking for confirmation when deleting items
- using success messages when users create new items
- providing them with helpful error messages when something goes wrong
- using commonly understood pagination controls on the app’s tables
Using these established patterns helps reduce the amount of new things that new users have to learn in order to use the app.
Back-end
Understanding the problem
The need to modernise and make the application more performant was top-of-mind during the re-write. It was no surprise that the app experienced frequent periods of downtimes, drop-offs and DNS resolution failures, given how out of date many of the libraries the app was using were.
So before we began the rewrite, we ran tests to establish a baseline of expectations. At this stage we debated which language to use – Golang or Rust. We decided to go with the former as it better suited our needs.
What we found in the baseline tests
To carry out our initial tests we built a simple version of the URL shortener redirect function of the application in the form of an API. This was important to the core design approach we chose. As well as establishing a baseline for performance, we used these tests to see how response times would be distributed, as well as get a feel for response time delays, overall performance under strain and general reliability.
To cut a long story short, we found the following:
- the Golang rewrite consistently outperformed the Ruby original in terms of response time and response delay.
- the Golang rewrite showed more consistent performance.
- while both applications appeared equally reliable, the Golang implementation handled higher loads more efficiently.
That last point is worth quantifying - the Golang version could handle 90 times more requests than the Ruby version, so not just more efficient, but a lot more efficient.
With our initial tests completed, we began the rewrite, splitting the application in two: first a frontend that was built using the NextJS framework, and the backend which was written using Golang.
The immediate challenges
One of the earliest challenges experienced was performance related, and a bit of a rookie mistake. Because of differences between development, staging, and production environments, we found that performance varied significantly between environments. This meant that while our application worked well on development, it suffered on the other environments. On further investigation, we discovered that the database had no optimisations to make query processing faster, so... we made some.
Another challenge that we encountered was an exponential performance drop-off with URL aliases. If we had a URL alias getting hit by several thousand requests, the application's request processing would get significantly slower. The solution to this problem was a matter of caching, and it allowed the application to process over 1.2 million requests per minute. That's an average of upwards of twenty thousand requests per second. To put this into perspective, we went from processing about 200,000 requests per minute to over six times that amount, simply by implementing proper caching strategies.
The database outage crisis
The final challenge occurred much later during development. We experienced a database outage and as a result, we lost access to the application and its redirect capacity. The application was calling the database, but there was no one home. To make matters worse, this was just before Open Day; a BIG problem because of how widespread the use of short URLs is during this period. If we couldn't get it back up and prevent this from happening again, we feared for the worst.
While we were waiting for the database to come back online, we were busy working on a solution to ensure we had some sort of layer to prevent complete cut-off from the database, should it go down again. After much research we decided that the best and most reasonable approach was to implement a middle layer sync service that added an embedded key/value database to the application.
For this, we chose bbolt, a key/value store based on Howard Chu's LMDB project.
This solution meant that we could:
- still create aliases during a database outage that would later be synced into the DB once a connection was established
- fetch alias-related data from the application
- create/destroy temporary caches of alias information
- sync all data accumulated during the outage
The service also has two redundancies: an active sync service that writes to a file, and an in-memory store. These help make the service more reliable.
All transactions are atomic to ensure that data cannot be lost or corrupted. When in this state, however, full CRUD (create, read, update, delete) only exists for devs, with everyone else only having read access. This approach means that even during complete database failures, our URL shortener can continue to function, albeit with limited write capabilities for non-developers.
What’s next?
The new application has been up and running for a little while now and is serving users well and meeting our expectations. During the design phase of the rebuild, we took a ‘minimum desirable product’ approach, and we knowingly held back on a few features that we wanted to introduce, to keep to a reasonable development time. Some of these include introducing a tagging system to make finding groups of aliases easier, as well as taking another look at the user interface for individual alias records to make it easier to see the history of an alias. In the near future, we want to iterate on the new URL shortener and hopefully introduce some of these new features.
Respond
