Digital Marketing & Communications

We've seen 1s and 0s you wouldn't believe

Topic: Development

How we implemented file uploading in our CMS

📥  CMS, Development

"dude-wheres-my-file

The setup

We have built into our CMS the ability to attach files to our Publication content type. Here I'll go cover the steps and hurdles we went through in implementing this very common feature.

On our first pass we took the often sensible approach of keeping scope small. We decided to build the ability to attach one file. Then we would iterate towards multiple attachments in later sprints.

Our first decision early on was to pick Thoughtbot's Paperclip library to help us implement this feature. Thoughtbot have a strong reputation among the Ruby community and we admire their approach of keeping things simple.

As part of our first iteration we had to do a ton of work around infrastructure set up too.

  • we needed dedicated network file storage for:
    • staging
    • training
    • and production
  • plumb in these file mounts to the CMS Editor so that we can show the file back to the user
  • changed our Mina deploy scripts to add an alias into RAILS_ROOT/public
  • and then copy the file on publish to the web server along with our HTML static page outputs

Simple!

The twist

After we shipped this first iteration we discovered a bug. Once you uploaded a file and hit save, if you encountered a Rails validation error you would lose your new attachment. So you'd fix your validation error but you wouldn't know that you had to upload that file again. This was especially bad if you were editing to change the attached file. If you had to fix other changes, you wouldn't realise that the attachment reverts to the original.

Twisting further

But Paperclip can handle this right? Wrong! It's a known bug (since 2009) that they won't fix! The most commonly suggested fix to this bug is to re-implement with an alternative library called CarrierWave.

A developer having a bit of a rage

Rage!

Act two: So CarrierWave...

...fixed this bug. It involved a rewrite of the code, but at least we could re-use all the infrastructure we'd set up. Next up, multiple attachments!

Another twist!

Our first attempt at implementing multiple attachments brought back the bug...

A developer having another rage

Double...

A developer having even more rage

...rage!

The final act

This time CarrierWave wasn't caching the attachment because we had applied it on the parent object in our polymorphic model. The fix was to re-do the code (again). Thankfully this time not as drastically as before. We are still using CarrierWave after all. We simply (!) applied CarrierWave to the Publication object itself rather than to its parent.

This model is closer to what we started with in our first implementation as well.

So now it's all simpler.

Still...

Despite all the hardship I was reminded that 10 years ago when we built the LMF we had to support file uploading. In that time we had to do a lot more of the work ourselves. Even though we had things like Apache Commons' IO library, all that gave us was binary streams and we still needed to write code to handle output to disk. That all took much, much longer than this rollercoaster attempt we made in Rails many years later. It's been an interesting contrast.

bath.ac.uk content by the numbers and the next steps

📥  Beta, Development, Style, content and design, Tools

In March, the Digital team set out on an ambitious project to inventory bath.ac.uk.

Our purpose was to learn more about the content we create and the publishers who write it. Gathering this knowledge with a thorough inventory process is something that I have wanted to do ever since I joined Bath in 2011.

This is what we found, how we found it and what our findings mean for how we plan, govern and build better content.
(more…)

 

Deploying Rails applications using Mina and Bamboo

📥  Development, Tools

We use Mina to deploy our Ruby on Rails projects. With our deployment scripts written and packaged into a gem we wanted to make use of our continuous integration server to build and deploy automatically to the production and staging environments.

Our continuous integration server is Bamboo which needed a little configuring to play nicely with Ruby. The packages containing the Ruby binaries were installed on the server and recognised by Bamboo (via auto-detection on the Server Capabilities admin screen) then we added the free Bamboo Ruby Plugin to give us Bundler tasks. This kept the deployment plans easy to understand by avoiding as much as possible the need for manual scripting.

The important tasks for us were "Bundler Install" (obvious what that does) and "Bundler CLI" which let us run our mina deployment commands using the bundler environment with "bundle exec". A bit of messing around with SSH keys and it all works beautifully.

The final setup is:

  • A Bamboo build plan pulls the code from github and makes it available as an artifact (tests are run here)
  • A Bamboo deployment plan takes the artifact, runs "bundle install" to get the code required for deployment then runs the mina tasks to push it to the server

Bamboo allows us to trigger each of these steps automatically so we can deploy a new version of our application just by merging code into the appropriate branch in our repositories. We deploy to both our staging and production environments in this way which makes for a simple workflow, all in Github. The results of the build are sent to us via email and appear in our Slack channel. Bamboo has also let us schedule a rebuild and redeploy of our staging environment on a monthly basis so we will be alerted if a piece of infrastructure has changed and caused our tests to start failing.

 

Automatically shipping early and often

📥  Beta, Development

We're an agile team, but rather than dogmatically follow a proscriptive capital-a Agile methodology we try to hew closely to the principles behind the Agile Manifesto.

Principle 3:

"Working software is delivered frequently (weeks rather than months)"

We're trying to do better than that. Sixty days after starting, we made the 100th release of our beta CMS to the staging server and the 21st release to our production server.

As Kelv has described before, "staging" is where features get deployed as and when they are completed so that we can run acceptance testing before they move into our production and training environments.

This little milestone represents many hundreds of individual changes by both developers and content writers over the last few months. Because each change is automatically sent to our servers when it's completed, this milestone happened completely transparently and with no fuss. This is exactly as it should be!

This is the first project where we've used Continuous Delivery. It did take some time to plan and set up the infrastructure in advance, and because we're still in development we're yet to battle-test the fact that the frequent deployments are totally seamless for people using the CMS. Nevertheless, the fact that we can deploy whenever we choose is a massive time-saver, allowing us to maintain a state of flow for sustained periods.

Some of our other projects are still using point-in-time pre-scheduled releases which we need to run manually and they already feel archaic. We're hopeful that we can take many of the processes we've developed during the CMS beta and apply them to these other applications, removing the burden of having to remember (or look up) many different deployment configuration intricacies.

 

Why static publishing and why Hugo?

📥  Beta, Development

As we've written before, we're going to try to use Hugo for statically publishing www.bath.ac.uk. There's a nice video showing it publishing 5,000 pages in under 7 seconds, and our testing gives similar numbers.

Static site serving has been big news over the past few years, mainly fuelled by the release of Jekyll. The problem for us is that we have many thousands of pages and many of these tools are written in Ruby or Node.js and are slow. If we want to change part of the site template then every page would need to be regenerated, and we want this to happen as quickly as possible.

At the moment there is almost no delay between making a change to the site and seeing it live on www.bath.ac.uk because we deliver pages straight from our database, but this means maintaining a more complicated and fragile infrastructure than we would like. Switching to a fast, statically published site will improve site stability and reduce our maintenance overhead, allowing us to spend more time on features.

Hugo is a young project and has some of the rough edges you might associate with that, but we're pretty hopeful that it will see us through the next period of innovation on the bath.ac.uk platform. If we do run into serious problems then we will probably switch to Jekyll until we were in a position to either scale out direct hosting from Rails or introduce a dedicated caching layer like Varnish.

 

A brief overview of the technical platform for alpha.bath.ac.uk

📥  Beta, Development

Our current CMS platform was introduced in 2007. It is a large, monolithic Java application which is hard to extend or customise. For the Alpha of a new CMS we wanted to base it on a popular web framework which would allow us to easily customise and extend a core platform, whilst also allowing us to use a broad range of third-party libraries.

We decided to base our Alpha on Rails, Contentful and Hugo. We used Rails to build a lightweight admin interface which changed our content in Contentful. Clicking 'publish' then sent a webhook notification to another Rails application we wrote which then used Hugo to regenerate the static HTML page. This general system design of loose coupling worked well and was a good introduction to the strengths and weaknesses of each of the tools we used.

We will keep Rails and Hugo as we move into the beta, but the libraries for interaction with Contentful were just too slow for us to use in real time. After we'd finished our Alpha a set of third-party libraries were released which might have helped, but nothing would be faster than connecting to a content store running here, and so that's what we'll do for the beta.

Deploying our Ruby on Rails applications

📥  Development, Tools

We are moving away from a Java based infrastructure to writing applications in Ruby on Rails. An important part of our former infrastructure was the ability to automatically deploy our software via the continuous integration server via a series of Ant tasks so we have been looking for a good way to do the same sort of thing with Rails code.

The established choice for Rails automated deployment seems to be Capistrano however there have been complaints about its speed and there seems to be a shift towards Mina which behaves differently under the hood.

Our intended development roadmap has us building a large number of small Rails projects all of which will need very similar deployment scripts so we also need a sensible way to manage the distribution of those scripts. We wrapped them in a gem which could simply be added to the project Gemfile and added a helper script which can set up a project and generate deployment new environments as needed.

We have made the gem public on our team github account with some instructions for use in the README and will be pushing updates as we add features and fix problems. If you find it useful, please let us know!

 

Using Rails

📥  Development

After ten years of Java and PHP, the Digital team have started writing all new applications, including our new CMS, in Ruby on Rails.

There are two reasons why now is a good time for us to do this. The first is that Rails is considered old hat, which means that it's now a tried and tested platform. The other is that improvements in our server virtualisation infrastructure mean that we no longer need to wait for Ruby packages to be available for our Solaris servers, because we can create new Linux servers as and when we need them. Solaris was a big blocker because the most recent Ruby package for Solaris is 1.8.7, which was released all the way back in 2008.

We are happy with our Java and PHP applications, and won't be looking to rewrite those in the near-term. We will continue to write applications in those languages where appropriate but are convinced that the Ruby community has shown itself to be strongly united behind Rails, making updates and community support much better than it is for the many competing Java and PHP frameworks.

Rails isn't perfect. It has regular security vulnerability announcements and then, assuming the patch is simple, we need to schedule the downtime to patch our applications. We have a channel in Slack which is subscribed to the rubyonrails-security mailing list and so we find out about new vulnerabilities quickly.

We're not currently planning on hosting public-facing applications with Rails, keeping them for internally-facing applications like the CMS editor. This will give us time to understand how to scale up our servers and applications appropriately and, since the CMS creates static HTML files it means we will be able to leave our powerful webservers doing the job they're already optimised for.

There's been a lot of learning involved in going from a blank-slate to a production-ready Rails application, and there'll be more to come, but we're happy with Rails and confident that it will serve us well in the coming years.

 

Our Github workflow

📥  Development

We are only 10 months into adopting Github Enterprise for our version control system and have many things still to learn. Near the start of that period we needed a workflow for managing the contributions that our team of devs, designers and editors make to our projects.

We decided to have a look at what was out there first and adapt something to our needs, taking into consideration our relative inexperience with Git in general.

Pick one

This helpful guide by Atlassian to some workflows was good at giving us an intial overview. We were able to discount the various workflows for either being to complicated for us (Gitflow) or too basic (Forking workflow).

We looked at the Github Workflow as well and found it suitable for our level of experience and how we wanted to work.

Change it

There are a couple of minor things we do differently though in our team. First of all we don't work against a branch called master.

What we have instead are staging and production branches. These reflect our platforms that we deploy to. staging is actually the default branch of all our project repos.

We always branch from staging to work on our features. Our Pull Requests are for merging back into staging. We then merge into production from staging. This last bit is still a bit clunky as we have to issue another PR for that merge.

Push it

Salt n Peppa - Push It

 

The staging branch for a project deploys automatically. Changes to repos send triggers from Github to Bamboo (using the built in Bamboo Service Hook) to run build and deploy plans. For production we'd rather have control over when this happens. So while builds are still automatic for production, we manually trigger deployments at the push of a button.

We are using mina to push our apps to our servers, executed by Bamboo. Bamboo has an official Ruby plugin which we are using. This includes running Bundler tasks.

More polish?

What we have so far is a little bit rough still on some edges (the PR to production for example), but overall we are finding that this workflow model is working well.

This is not to say that as we gain more experience we won't revisit the various models of workflow in the future.

 

Alpha site release notes, 12 - 19 January

📥  Beta, Communication, Development, Style, content and design

On Monday 12 January, we launched an alpha version of our site to internal users, including a new external homepage and the about section, which is controlled by our new CMS alpha. Collectively we've been referring to these as the Alpha site.

The Alpha site is still a work in progress, and we're testing it with our internal users now because your feedback will help to shape its development. We want to see how you use it, whether it meets your needs as a user, and what problems you encounter — so we can fix them before they wreak havoc on a larger audience.

What we did this week

We've had a lot of useful feedback from our users already, and we've fixed some of the issues you've raised, as well as continuing to tackle pre-planned work from the Alpha backlog.

  • Fixed issues that meant some maps were difficult to find or unavailable, including updating redirects from old URLs and changing which page included the map links.
  • The homepage is now served statically, which will lead to performance improvements and better stability if the CMS is unavailable.
  • Improved how the lead image on the homepage displays on smaller screens. A much larger proportion of the image is now visible and editors can select the alignment for how the image is cropped.
Screencap of the homepage Alpha.

Image cropping on small screens before.

Screencap of the homepage Alpha.

Image cropping on small screens after.

  • Worked on a first iteration of an editorial calendar for the homepage to ensure it's frequently updated with interesting, relevant content.
  • Replaced the National Student Survey graphic in the homepage footer with a better-looking SVG version.
  • Standardised design elements in the header and search bar across the Alpha site.
  • Reordered the link lists on the about landing page so it now leads with our mission and values.
  • Changed the wording that links to location pages on team overview pages from "The team is located in [location page]." to the more flexible "Find out more about this team's location in [location page]."

Keep it coming

Please continue to share your feedback with us. Your feedback is most helpful when it's specific - let us know what task you wanted to accomplish on the website and what stopped you, and we'll look into how we can fix it.