Digital Marketing & Communications

We've seen 1s and 0s you wouldn't believe

Topic: Development

Launching Worldwide

📥  Development, International

We have recently launched Worldwide on bath.ac.uk.

It is a new central hub to curate international content across the website and will focus on a specific country or region. This is to support the implementation of the University’s International Strategy.

For the first edition, the page aggregates relevant content items to showcase the University’s activities and impact in South Africa. The goal is to send out clear messages about how our work in South Africa provides solutions to global environmental and social challenges, empowers future higher education leaders and develops top athletes of tomorrow.

First of its kind

This is the first time we have experimented to present the University's internationalisation by:

  • featuring stories with a content marketing approach, focusing on creating and distributing valuable, relevant and consistent content to target clearly-defined users
  • using featured content to showcase the University's impact in various fields - from research to student experience to sports
  • producing content on the University’s overseas activities with coordination between faculties and professional services
  • using the Collection content type in the new CMS to curate content items
  • intensively promoting content on social media targeting specific user groups during the post-launch period

We learn as we go

There has been so much to learn throughout the entire process of delivering Worldwide, from content creation to curation. We will continue working in an agile way and embracing a culture of constantly learning and iterating through each round of producing Worldwide.

What tech we use to test our CMS: Minitest and Capybara

📥  Development, Tools

tumblr_lmht06jDfG1qhed3yo1_500

The road less travelled

For the Editor side of our new CMS, we have made a couple of choices in testing that are a little bit off the beaten track.

For starters we are using Minitest instead of RSpec. Most Rails devs will pick RSpec as that's what comes out of the box. And Rails devs who have been around for a few years are already familiar with RSpec.

tumblr_mg57quMO7h1qj8u1do1_540

You say shoulda, we say assert

We favoured Minitest because of the low barrier to entry it provided us. As a dev house that had mostly written apps in Java in the past, we're already comfortable with unit tests. We're also already familiar with the assertion style. We felt that as a new DSL to learn it's a much smaller and easier set to absorb than compared to RSpec.

On top of that we're using Capybara to run functional tests. This is yet another DSL to learn. We could've overwhelmed ourselves with DSLs! This is again written in Minitest assertions.

This is not without any downsides.

With RSpec and specs being the most common approach, examples and documentation are almost all written in that way. So we still have to negotiate the nuances of spec tests when reading examples. But because we're still unfamiliar to spec tests they're slightly impenetrable. On balance we've lowered the barrier to adoption and got to shipping code quicker. We were already on a learning curve with Rails itself so the early win was worth it.

The wider picture

With our stack we have 2 discrete services, glued together by Redis. Our current problem is that we only have tests for the CMS Editor and not the end to end publishing process. We do have some unit tests on the CMS Publisher middle layer. This is the bit that receives data from the CMS Editor (via Redis) on publish (or preview) and calls Hugo to render out our pages as static HTML. What we want is a way to test the functionality from the point of creating a new piece of content all the way to rendering it out as a final piece of HTML. Currently the impact to the front end from changes we make on the CMS Editor (for example) are obscured from us.

What we'd like to do is make use of something like Docker and auto deploy all the apps via our CI server Bamboo. We'd then have an end to end containerised system we could run full functional tests against. We'd also end up with the front end outputs which we could run regression tests on.

This is all stuff we plan for the future. Hopefully the next time I post we'll have achieved some of that!

How we implemented file uploading in our CMS

📥  CMS, Development

"dude-wheres-my-file

The setup

We have built into our CMS the ability to attach files to our Publication content type. Here I'll go cover the steps and hurdles we went through in implementing this very common feature.

On our first pass we took the often sensible approach of keeping scope small. We decided to build the ability to attach one file. Then we would iterate towards multiple attachments in later sprints.

Our first decision early on was to pick Thoughtbot's Paperclip library to help us implement this feature. Thoughtbot have a strong reputation among the Ruby community and we admire their approach of keeping things simple.

As part of our first iteration we had to do a ton of work around infrastructure set up too.

  • we needed dedicated network file storage for:
    • staging
    • training
    • and production
  • plumb in these file mounts to the CMS Editor so that we can show the file back to the user
  • changed our Mina deploy scripts to add an alias into RAILS_ROOT/public
  • and then copy the file on publish to the web server along with our HTML static page outputs

Simple!

The twist

After we shipped this first iteration we discovered a bug. Once you uploaded a file and hit save, if you encountered a Rails validation error you would lose your new attachment. So you'd fix your validation error but you wouldn't know that you had to upload that file again. This was especially bad if you were editing to change the attached file. If you had to fix other changes, you wouldn't realise that the attachment reverts to the original.

Twisting further

But Paperclip can handle this right? Wrong! It's a known bug (since 2009) that they won't fix! The most commonly suggested fix to this bug is to re-implement with an alternative library called CarrierWave.

A developer having a bit of a rage

Rage!

Act two: So CarrierWave...

...fixed this bug. It involved a rewrite of the code, but at least we could re-use all the infrastructure we'd set up. Next up, multiple attachments!

Another twist!

Our first attempt at implementing multiple attachments brought back the bug...

A developer having another rage

Double...

A developer having even more rage

...rage!

The final act

This time CarrierWave wasn't caching the attachment because we had applied it on the parent object in our polymorphic model. The fix was to re-do the code (again). Thankfully this time not as drastically as before. We are still using CarrierWave after all. We simply (!) applied CarrierWave to the Publication object itself rather than to its parent.

This model is closer to what we started with in our first implementation as well.

So now it's all simpler.

Still...

Despite all the hardship I was reminded that 10 years ago when we built the LMF we had to support file uploading. In that time we had to do a lot more of the work ourselves. Even though we had things like Apache Commons' IO library, all that gave us was binary streams and we still needed to write code to handle output to disk. That all took much, much longer than this rollercoaster attempt we made in Rails many years later. It's been an interesting contrast.

bath.ac.uk content by the numbers and the next steps

📥  Beta, Development, Style, content and design, Tools

In March, the Digital team set out on an ambitious project to inventory bath.ac.uk.

Our purpose was to learn more about the content we create and the publishers who write it. Gathering this knowledge with a thorough inventory process is something that I have wanted to do ever since I joined Bath in 2011.

This is what we found, how we found it and what our findings mean for how we plan, govern and build better content.
(more…)

 

Deploying Rails applications using Mina and Bamboo

📥  Development, Tools

We use Mina to deploy our Ruby on Rails projects. With our deployment scripts written and packaged into a gem we wanted to make use of our continuous integration server to build and deploy automatically to the production and staging environments.

Our continuous integration server is Bamboo which needed a little configuring to play nicely with Ruby. The packages containing the Ruby binaries were installed on the server and recognised by Bamboo (via auto-detection on the Server Capabilities admin screen) then we added the free Bamboo Ruby Plugin to give us Bundler tasks. This kept the deployment plans easy to understand by avoiding as much as possible the need for manual scripting.

The important tasks for us were "Bundler Install" (obvious what that does) and "Bundler CLI" which let us run our mina deployment commands using the bundler environment with "bundle exec". A bit of messing around with SSH keys and it all works beautifully.

The final setup is:

  • A Bamboo build plan pulls the code from github and makes it available as an artifact (tests are run here)
  • A Bamboo deployment plan takes the artifact, runs "bundle install" to get the code required for deployment then runs the mina tasks to push it to the server

Bamboo allows us to trigger each of these steps automatically so we can deploy a new version of our application just by merging code into the appropriate branch in our repositories. We deploy to both our staging and production environments in this way which makes for a simple workflow, all in Github. The results of the build are sent to us via email and appear in our Slack channel. Bamboo has also let us schedule a rebuild and redeploy of our staging environment on a monthly basis so we will be alerted if a piece of infrastructure has changed and caused our tests to start failing.

 

Automatically shipping early and often

📥  Beta, Development

We're an agile team, but rather than dogmatically follow a proscriptive capital-a Agile methodology we try to hew closely to the principles behind the Agile Manifesto.

Principle 3:

"Working software is delivered frequently (weeks rather than months)"

We're trying to do better than that. Sixty days after starting, we made the 100th release of our beta CMS to the staging server and the 21st release to our production server.

As Kelv has described before, "staging" is where features get deployed as and when they are completed so that we can run acceptance testing before they move into our production and training environments.

This little milestone represents many hundreds of individual changes by both developers and content writers over the last few months. Because each change is automatically sent to our servers when it's completed, this milestone happened completely transparently and with no fuss. This is exactly as it should be!

This is the first project where we've used Continuous Delivery. It did take some time to plan and set up the infrastructure in advance, and because we're still in development we're yet to battle-test the fact that the frequent deployments are totally seamless for people using the CMS. Nevertheless, the fact that we can deploy whenever we choose is a massive time-saver, allowing us to maintain a state of flow for sustained periods.

Some of our other projects are still using point-in-time pre-scheduled releases which we need to run manually and they already feel archaic. We're hopeful that we can take many of the processes we've developed during the CMS beta and apply them to these other applications, removing the burden of having to remember (or look up) many different deployment configuration intricacies.

 

Why static publishing and why Hugo?

📥  Beta, Development

As we've written before, we're going to try to use Hugo for statically publishing www.bath.ac.uk. There's a nice video showing it publishing 5,000 pages in under 7 seconds, and our testing gives similar numbers.

Static site serving has been big news over the past few years, mainly fuelled by the release of Jekyll. The problem for us is that we have many thousands of pages and many of these tools are written in Ruby or Node.js and are slow. If we want to change part of the site template then every page would need to be regenerated, and we want this to happen as quickly as possible.

At the moment there is almost no delay between making a change to the site and seeing it live on www.bath.ac.uk because we deliver pages straight from our database, but this means maintaining a more complicated and fragile infrastructure than we would like. Switching to a fast, statically published site will improve site stability and reduce our maintenance overhead, allowing us to spend more time on features.

Hugo is a young project and has some of the rough edges you might associate with that, but we're pretty hopeful that it will see us through the next period of innovation on the bath.ac.uk platform. If we do run into serious problems then we will probably switch to Jekyll until we were in a position to either scale out direct hosting from Rails or introduce a dedicated caching layer like Varnish.

 

A brief overview of the technical platform for alpha.bath.ac.uk

📥  Beta, Development

Our current CMS platform was introduced in 2007. It is a large, monolithic Java application which is hard to extend or customise. For the Alpha of a new CMS we wanted to base it on a popular web framework which would allow us to easily customise and extend a core platform, whilst also allowing us to use a broad range of third-party libraries.

We decided to base our Alpha on Rails, Contentful and Hugo. We used Rails to build a lightweight admin interface which changed our content in Contentful. Clicking 'publish' then sent a webhook notification to another Rails application we wrote which then used Hugo to regenerate the static HTML page. This general system design of loose coupling worked well and was a good introduction to the strengths and weaknesses of each of the tools we used.

We will keep Rails and Hugo as we move into the beta, but the libraries for interaction with Contentful were just too slow for us to use in real time. After we'd finished our Alpha a set of third-party libraries were released which might have helped, but nothing would be faster than connecting to a content store running here, and so that's what we'll do for the beta.

Deploying our Ruby on Rails applications

📥  Development, Tools

We are moving away from a Java based infrastructure to writing applications in Ruby on Rails. An important part of our former infrastructure was the ability to automatically deploy our software via the continuous integration server via a series of Ant tasks so we have been looking for a good way to do the same sort of thing with Rails code.

The established choice for Rails automated deployment seems to be Capistrano however there have been complaints about its speed and there seems to be a shift towards Mina which behaves differently under the hood.

Our intended development roadmap has us building a large number of small Rails projects all of which will need very similar deployment scripts so we also need a sensible way to manage the distribution of those scripts. We wrapped them in a gem which could simply be added to the project Gemfile and added a helper script which can set up a project and generate deployment new environments as needed.

We have made the gem public on our team github account with some instructions for use in the README and will be pushing updates as we add features and fix problems. If you find it useful, please let us know!

 

Using Rails

📥  Development

After ten years of Java and PHP, the Digital team have started writing all new applications, including our new CMS, in Ruby on Rails.

There are two reasons why now is a good time for us to do this. The first is that Rails is considered old hat, which means that it's now a tried and tested platform. The other is that improvements in our server virtualisation infrastructure mean that we no longer need to wait for Ruby packages to be available for our Solaris servers, because we can create new Linux servers as and when we need them. Solaris was a big blocker because the most recent Ruby package for Solaris is 1.8.7, which was released all the way back in 2008.

We are happy with our Java and PHP applications, and won't be looking to rewrite those in the near-term. We will continue to write applications in those languages where appropriate but are convinced that the Ruby community has shown itself to be strongly united behind Rails, making updates and community support much better than it is for the many competing Java and PHP frameworks.

Rails isn't perfect. It has regular security vulnerability announcements and then, assuming the patch is simple, we need to schedule the downtime to patch our applications. We have a channel in Slack which is subscribed to the rubyonrails-security mailing list and so we find out about new vulnerabilities quickly.

We're not currently planning on hosting public-facing applications with Rails, keeping them for internally-facing applications like the CMS editor. This will give us time to understand how to scale up our servers and applications appropriately and, since the CMS creates static HTML files it means we will be able to leave our powerful webservers doing the job they're already optimised for.

There's been a lot of learning involved in going from a blank-slate to a production-ready Rails application, and there'll be more to come, but we're happy with Rails and confident that it will serve us well in the coming years.