Digital Marketing & Communications

We've seen 1s and 0s you wouldn't believe

Topic: Development

Deploying Rails applications using Mina and Bamboo

📥  Development, Tools

We use Mina to deploy our Ruby on Rails projects. With our deployment scripts written and packaged into a gem we wanted to make use of our continuous integration server to build and deploy automatically to the production and staging environments.

Our continuous integration server is Bamboo which needed a little configuring to play nicely with Ruby. The packages containing the Ruby binaries were installed on the server and recognised by Bamboo (via auto-detection on the Server Capabilities admin screen) then we added the free Bamboo Ruby Plugin to give us Bundler tasks. This kept the deployment plans easy to understand by avoiding as much as possible the need for manual scripting.

The important tasks for us were "Bundler Install" (obvious what that does) and "Bundler CLI" which let us run our mina deployment commands using the bundler environment with "bundle exec". A bit of messing around with SSH keys and it all works beautifully.

The final setup is:

  • A Bamboo build plan pulls the code from github and makes it available as an artifact (tests are run here)
  • A Bamboo deployment plan takes the artifact, runs "bundle install" to get the code required for deployment then runs the mina tasks to push it to the server

Bamboo allows us to trigger each of these steps automatically so we can deploy a new version of our application just by merging code into the appropriate branch in our repositories. We deploy to both our staging and production environments in this way which makes for a simple workflow, all in Github. The results of the build are sent to us via email and appear in our Slack channel. Bamboo has also let us schedule a rebuild and redeploy of our staging environment on a monthly basis so we will be alerted if a piece of infrastructure has changed and caused our tests to start failing.


Automatically shipping early and often

📥  Beta, Development

We're an agile team, but rather than dogmatically follow a proscriptive capital-a Agile methodology we try to hew closely to the principles behind the Agile Manifesto.

Principle 3:

"Working software is delivered frequently (weeks rather than months)"

We're trying to do better than that. Sixty days after starting, we made the 100th release of our beta CMS to the staging server and the 21st release to our production server.

As Kelv has described before, "staging" is where features get deployed as and when they are completed so that we can run acceptance testing before they move into our production and training environments.

This little milestone represents many hundreds of individual changes by both developers and content writers over the last few months. Because each change is automatically sent to our servers when it's completed, this milestone happened completely transparently and with no fuss. This is exactly as it should be!

This is the first project where we've used Continuous Delivery. It did take some time to plan and set up the infrastructure in advance, and because we're still in development we're yet to battle-test the fact that the frequent deployments are totally seamless for people using the CMS. Nevertheless, the fact that we can deploy whenever we choose is a massive time-saver, allowing us to maintain a state of flow for sustained periods.

Some of our other projects are still using point-in-time pre-scheduled releases which we need to run manually and they already feel archaic. We're hopeful that we can take many of the processes we've developed during the CMS beta and apply them to these other applications, removing the burden of having to remember (or look up) many different deployment configuration intricacies.


Why static publishing and why Hugo?

📥  Beta, Development

As we've written before, we're going to try to use Hugo for statically publishing There's a nice video showing it publishing 5,000 pages in under 7 seconds, and our testing gives similar numbers.

Static site serving has been big news over the past few years, mainly fuelled by the release of Jekyll. The problem for us is that we have many thousands of pages and many of these tools are written in Ruby or Node.js and are slow. If we want to change part of the site template then every page would need to be regenerated, and we want this to happen as quickly as possible.

At the moment there is almost no delay between making a change to the site and seeing it live on because we deliver pages straight from our database, but this means maintaining a more complicated and fragile infrastructure than we would like. Switching to a fast, statically published site will improve site stability and reduce our maintenance overhead, allowing us to spend more time on features.

Hugo is a young project and has some of the rough edges you might associate with that, but we're pretty hopeful that it will see us through the next period of innovation on the platform. If we do run into serious problems then we will probably switch to Jekyll until we were in a position to either scale out direct hosting from Rails or introduce a dedicated caching layer like Varnish.


A brief overview of the technical platform for

📥  Beta, Development

Our current CMS platform was introduced in 2007. It is a large, monolithic Java application which is hard to extend or customise. For the Alpha of a new CMS we wanted to base it on a popular web framework which would allow us to easily customise and extend a core platform, whilst also allowing us to use a broad range of third-party libraries.

We decided to base our Alpha on Rails, Contentful and Hugo. We used Rails to build a lightweight admin interface which changed our content in Contentful. Clicking 'publish' then sent a webhook notification to another Rails application we wrote which then used Hugo to regenerate the static HTML page. This general system design of loose coupling worked well and was a good introduction to the strengths and weaknesses of each of the tools we used.

We will keep Rails and Hugo as we move into the beta, but the libraries for interaction with Contentful were just too slow for us to use in real time. After we'd finished our Alpha a set of third-party libraries were released which might have helped, but nothing would be faster than connecting to a content store running here, and so that's what we'll do for the beta.

Deploying our Ruby on Rails applications

📥  Development, Tools

We are moving away from a Java based infrastructure to writing applications in Ruby on Rails. An important part of our former infrastructure was the ability to automatically deploy our software via the continuous integration server via a series of Ant tasks so we have been looking for a good way to do the same sort of thing with Rails code.

The established choice for Rails automated deployment seems to be Capistrano however there have been complaints about its speed and there seems to be a shift towards Mina which behaves differently under the hood.

Our intended development roadmap has us building a large number of small Rails projects all of which will need very similar deployment scripts so we also need a sensible way to manage the distribution of those scripts. We wrapped them in a gem which could simply be added to the project Gemfile and added a helper script which can set up a project and generate deployment new environments as needed.

We have made the gem public on our team github account with some instructions for use in the README and will be pushing updates as we add features and fix problems. If you find it useful, please let us know!


Using Rails

📥  Development

After ten years of Java and PHP, the Digital team have started writing all new applications, including our new CMS, in Ruby on Rails.

There are two reasons why now is a good time for us to do this. The first is that Rails is considered old hat, which means that it's now a tried and tested platform. The other is that improvements in our server virtualisation infrastructure mean that we no longer need to wait for Ruby packages to be available for our Solaris servers, because we can create new Linux servers as and when we need them. Solaris was a big blocker because the most recent Ruby package for Solaris is 1.8.7, which was released all the way back in 2008.

We are happy with our Java and PHP applications, and won't be looking to rewrite those in the near-term. We will continue to write applications in those languages where appropriate but are convinced that the Ruby community has shown itself to be strongly united behind Rails, making updates and community support much better than it is for the many competing Java and PHP frameworks.

Rails isn't perfect. It has regular security vulnerability announcements and then, assuming the patch is simple, we need to schedule the downtime to patch our applications. We have a channel in Slack which is subscribed to the rubyonrails-security mailing list and so we find out about new vulnerabilities quickly.

We're not currently planning on hosting public-facing applications with Rails, keeping them for internally-facing applications like the CMS editor. This will give us time to understand how to scale up our servers and applications appropriately and, since the CMS creates static HTML files it means we will be able to leave our powerful webservers doing the job they're already optimised for.

There's been a lot of learning involved in going from a blank-slate to a production-ready Rails application, and there'll be more to come, but we're happy with Rails and confident that it will serve us well in the coming years.


Our Github workflow

📥  Development

We are only 10 months into adopting Github Enterprise for our version control system and have many things still to learn. Near the start of that period we needed a workflow for managing the contributions that our team of devs, designers and editors make to our projects.

We decided to have a look at what was out there first and adapt something to our needs, taking into consideration our relative inexperience with Git in general.

Pick one

This helpful guide by Atlassian to some workflows was good at giving us an intial overview. We were able to discount the various workflows for either being to complicated for us (Gitflow) or too basic (Forking workflow).

We looked at the Github Workflow as well and found it suitable for our level of experience and how we wanted to work.

Change it

There are a couple of minor things we do differently though in our team. First of all we don't work against a branch called master.

What we have instead are staging and production branches. These reflect our platforms that we deploy to. staging is actually the default branch of all our project repos.

We always branch from staging to work on our features. Our Pull Requests are for merging back into staging. We then merge into production from staging. This last bit is still a bit clunky as we have to issue another PR for that merge.

Push it

Salt n Peppa - Push It


The staging branch for a project deploys automatically. Changes to repos send triggers from Github to Bamboo (using the built in Bamboo Service Hook) to run build and deploy plans. For production we'd rather have control over when this happens. So while builds are still automatic for production, we manually trigger deployments at the push of a button.

We are using mina to push our apps to our servers, executed by Bamboo. Bamboo has an official Ruby plugin which we are using. This includes running Bundler tasks.

More polish?

What we have so far is a little bit rough still on some edges (the PR to production for example), but overall we are finding that this workflow model is working well.

This is not to say that as we gain more experience we won't revisit the various models of workflow in the future.


Alpha site release notes, 12 - 19 January

📥  Beta, Communication, Development, Style, content and design

On Monday 12 January, we launched an alpha version of our site to internal users, including a new external homepage and the about section, which is controlled by our new CMS alpha. Collectively we've been referring to these as the Alpha site.

The Alpha site is still a work in progress, and we're testing it with our internal users now because your feedback will help to shape its development. We want to see how you use it, whether it meets your needs as a user, and what problems you encounter — so we can fix them before they wreak havoc on a larger audience.

What we did this week

We've had a lot of useful feedback from our users already, and we've fixed some of the issues you've raised, as well as continuing to tackle pre-planned work from the Alpha backlog.

  • Fixed issues that meant some maps were difficult to find or unavailable, including updating redirects from old URLs and changing which page included the map links.
  • The homepage is now served statically, which will lead to performance improvements and better stability if the CMS is unavailable.
  • Improved how the lead image on the homepage displays on smaller screens. A much larger proportion of the image is now visible and editors can select the alignment for how the image is cropped.
Screencap of the homepage Alpha.

Image cropping on small screens before.

Screencap of the homepage Alpha.

Image cropping on small screens after.

  • Worked on a first iteration of an editorial calendar for the homepage to ensure it's frequently updated with interesting, relevant content.
  • Replaced the National Student Survey graphic in the homepage footer with a better-looking SVG version.
  • Standardised design elements in the header and search bar across the Alpha site.
  • Reordered the link lists on the about landing page so it now leads with our mission and values.
  • Changed the wording that links to location pages on team overview pages from "The team is located in [location page]." to the more flexible "Find out more about this team's location in [location page]."

Keep it coming

Please continue to share your feedback with us. Your feedback is most helpful when it's specific - let us know what task you wanted to accomplish on the website and what stopped you, and we'll look into how we can fix it.



📥  Beta, CMS, Development, Tools

We've changed just about everything to do with how we manage for the alpha, and this includes the underlying technology.

The live is hosted dynamically from a monolithic Java-based CMS (OpenCms) running on Solaris servers with Oracle as a back-end database and all the code is in Subversion.

The alpha site is being served from static HTML files generated by Hugo. The inputs are provided by multiple Ruby on Rails apps running on Linux servers using MySQL, MongoDB and an off-site service for data storage, and all the code is in our local GitHub Enterprise installation and is deployed from Bamboo by Mina.

The difference could not really be much bigger! It's certainly been a challenge to get up to speed with so many new things all at once but we think that this stack, or some version of it - that's what alphas are for! - should be more future-friendly than our existing infrastructure.

We'll go into more detail on how we're using each piece of technology in the following days and weeks.

Show & Tell, November 21 2014

📥  Communication, Design, Development, Show & Tell

Back in our spiritual home after the impromtu reshuffle that made our last Show & Tell session so special, we had a full roster of presenters and a diverse range of topics.

Ruby Idioms - Kelvin

Our developers are working more and more with Ruby — Rails in particular — and Kelvin has been challenged with providing instruction and direction to the team on the subtleties of how we should write Ruby differently from Java and PHP (other previous go-to production languages).

Far too much to cover in five minutes, we instead had a whistle-stop tour of the top ten seven things to be aware of, from not using unless statements with an else block, to replacing do...end blocks with curly braces if they are a single line.

A full rundown can be found in Kelv's github repo:

Less stuff - Dan

Still with me? Excellent. Next up was Dan, who talked us through taking a pragmatic approach to webfonts to provide a better user experience. A large part of the work was reducing the filesize of the font manifest file by 66% - theoretically providing a significantly improved loading time for those viewing the website on slow internet connections. The key was looking at the different weights of font being served by default, and making careful design choices that allowed us to provide maximum clarity and aesthetic with the minimum variety of styles and weights.

Dan then went on to propose a manifesto of using less as a starting point for design - tying in aspects of user-centered design, progressive enhancement, the mobile-first approach, and our existing delivery principles.

What do I do? - Katrina

Six months into her new post as Research Marketing Manager, guest speaker Katrina gave us the lowdown on what her job entails. It turns out that a fair amount of it is commercially sensitive, so I'll be skipping over that - no secrets for you.

Katrina spends a large amount of her time planning and coordinating large-scale campaigns to cement relations with University stakeholders. Currently we are tapping into the large amount of water-themed research that our academics are involved in and Katrina is putting the finishing touches to a six-month campaign relating to this.

When not devising ways to get our research the recognition it deserves, Katrina acts as a single point of contact between our academics and the various marketing teams that exist on campus at all different levels - from research teams, through departmental and faculty right up to the University Marketing and Comms. This aspect of her role has been extremely well-received on campus, as busy professors delight in having one single consistent person to deal with concerning their marketing.

The tale of BrowserStack - Tom

Continuing his series of talks concerning security, Tom Natt used the real world example of the recent attack on BrowserStack to illustrate what can happen when things go wrong.

Essentially, BrowserStack had an old computer that nobody used or maintained but was still connected to their network. A hacker discovered this and used the Shellshock vulnerability to take control and gain access to the API key for their AWS (Amazon Web Storage). From this they discovered the database password and attempted to download their entire customer database. This was when BrowserStack became aware of the hack and acted quickly to shut them down. It is still reckoned that 1% (approximately 5000 users) of the database was compromised.

We were about to use BrowserStack to assist us with some work, so this attack and the way that BrowserStack handled it (in terms of securing their system and managing their public profile) went a long way to reassuring us that they were still a suitable partner. Tom also made the point that having just been hacked, they were likely to be awake to the danger right now because of their recent experiences.

Alpha update - Ross

Our the last seven weeks the team has been working on several alphas (CMS, homepage, events, prospectus) and these have all been presented to members of our Digital Steering Group - which is comprised of almost all of our pro-vice-chancellors as well as the movers and shakers in senior management. The feedback has been overwhelmingly positive, with very enthusiastic engagement with aspects of the homepage and the CMS in particular (our most 'mature' alphas). Our plan was always to get the new homepage and areas of the site controlled by the new CMS in front of staff and students as soon as possible, and with the full support of the DSG, we are looking to do this in December.