If you set out to change something, you\u2019ll be asked to measure the change made. That\u2019s been our experience since we began our work creating a positive culture of public engagement with research over the last nine years. Our approach has been dynamic since 2012 and here I wanted to share some of the ways we\u2019ve evaluated culture change.<\/p>\n
The Public Engagement Unit<\/a> was formed as part of the RCUK (now UKRI) Public Engagement with Research Catalysts<\/a> funded culture change project Engaged360@Bath (2012-2015)<\/a><\/em>. When I am talking about our work to colleagues, nearly the first question I get asked is: \u201chow do you measure culture change?\u201d. I\u2019m not sure I have a pithy answer for this one, apart from saying that it\u2019s hard! This post is a reflection on how we\u2019ve evaluated culture change at the University of Bath and is some of the ways that have worked for us, but I\u2019d love to hear others\u2019 views and perspectives on their experiences of evidencing culture change in their organisation. Do feel free to drop me a line<\/a> and let's compare notes.<\/p>\n Our evaluation approach has evolved over time. This has been in response to the changing environment and drivers around us and the stage we'd reached on our culture change journey. These represent key moments that have shaped our approach to evaluation.<\/p>\n Our Engaged360@Bath<\/em> project came with reporting requirements which included an evaluation of what we had achieved over three years. We used three tools to help frame our work.<\/p>\n So, we had these three tools: the logic model, complexity model, and the EDGE tool, which helped us understand what we could evaluate. The EDGE tool helped inform where we would need to put interventions in place, and the logic model helped us define these interventions and the expected outputs we could report on. The complexity framing meant that we needed to keep our eyes open to emergent evidence of change, but that it would be best to focus on the interventions that we had control over.<\/p>\n The key to our overall approach in this initial phase of our work was adopting a mindset of reflection. It helped us in several ways. It was essential to the successful use of a complexity model as we had to keep an eye out for emergent learning as a result of our interventions, it also meant that we never had to shoe-horn evaluation into our work afterwards. We reflected on our meetings, were alive to indicators such as the language used by colleagues to describe engagement and shared our thinking with colleagues via blogs. It was simply how we did our work. It also provided an ongoing body of evidence that we could use to inform decisions such as whether to start, continue, tweak, or stop an activity. It also created a bank of resources that were informed by our colleagues\u2019 experiences based on the University of Bath context, rather than our own practitioner experiences from other settings. One example of this is our Top Tips guide<\/a> which we synthesised from the end of project reports that our colleagues submitted after they received funding from us for pilot work.<\/p>\n We produced an end of project report<\/a>, which included a full-scale evaluation. The evaluation reported against the objectives and project plan that had been proposed at the beginning. It wasn\u2019t really an evaluation of culture change because you can\u2019t affect long term change within three years. We could only provide evidence that we\u2019d put things in place (created the conditions for change), and that we had started the journey.<\/p>\n As well as producing our evaluation and end of project reports for funders, we were also making the case, as a Unit, for continued financial support from the University. We ended up having to do this twice. First as an extension of three years to sustain the work we\u2019d started in the Engaged360@Bath<\/em> project and a second time for core funding in 2018. These cases were made using the ongoing monitoring and evaluation data we were collecting about our work such as the number of researchers supported with grant writing or attending training and placing them in the broader institutional and sector context.<\/p>\n We also commissioned an external evaluator to talk with all the people we\u2019d directly engaged with over the years. This helped us understand and articulate our value, whilst also showing that our approach was working. It also added credibility to our claims.<\/p>\n Now, nine years after we began, our programme is reasonably settled, it is a programme that emerged during the initial interventions of our culture change work. Using the model of complexity theory meant that while during the beginning phases of our work we couldn\u2019t exactly predict what a culture of public engagement with research would look like, we can point at it now and say that 'this is what it looks like'. It also reassured us that the things that have stayed in place are robust and the things that have dropped by the wayside weren\u2019t the right thing at the right time. We use a tool called Outnav<\/a> to store the routine monitoring data we collect on our work as well as evaluation reports from our initiatives and it helps us produce reports. Outnav is based on a logic model approach so it works well for us.<\/p>\n So, how can we be confident about our culture of public engagement?<\/p>\n I mentioned the EDGE tool earlier. This is a self-assessment tool and we have progressed nicely along with it. An external measure like the EDGE tool is useful because it adds weight to our claims. However, there are other indicators. These are the things we couldn\u2019t predict. Faculties and Departments creating new roles (or partial roles) with responsibility for public engagement. Colleagues across all job families being able to confidently talk about and share learning on public engagement in our absence and colleagues incorporating meaningful public engagement into research proposals and programmes.<\/p>\n Culture change never ends. Yes, at the beginning there is a rapid, dramatic change, but a university is a dynamic organisation operating in a dynamic system, so we have to be constantly on the move and responsive to the organisational need. We can only do that by being perpetually vigilant and using evaluation and evidence to argue the case for change (or not) and keep aligning with strategic goals \/ influencing strategy development to include your agenda.<\/p>\n Being nine years on from our starting point and recognising change can take 10 years or so, we decided to reflect on our early approaches and produced a guide about culture change at the University of Bath<\/a>.<\/p>\n This habit of writing about our learning and keeping and active blog has been a key part of our reflective practice which has also helped our profile with our peers.<\/p>\n There are two issues we're currently thinking about at the minute:<\/p>\n If you're tackling evaluating culture change here are some things that worked for us:<\/p>\nOur approach to evaluating culture change<\/strong><\/h2>\n
Driver one: project reporting<\/strong><\/h3>\n
\n
Reflective practice<\/h4>\n
End of project report<\/h4>\n
Driver two: making the case for funding<\/strong><\/h3>\n
Driver three: routine monitoring and evaluation<\/strong><\/h3>\n
Where are we now?<\/strong><\/h2>\n
Issues we\u2019re thinking about at the moment <\/strong><\/h2>\n
\n
\u00a0<\/strong>Top tips<\/strong><\/h2>\n