Exploring the potential benefits and risks of ESD indicators

As I noted earlier, on Monday, I'm giving a talk on the potential benefits and risks of ESD indicators at a symposium at the University of Bern which explores an ESD indicator set  that has been generated by researchers and practitioners in Austria, Germany and Switzerland.  The outline of my talk is set out here:

Exploring the potential benefits and risks of ESD indicators

I’ve been asked to give a general introduction to the potential benefits and risks of ESD indicators.  I’ll begin by saying something about indicators in general, and about sustainable development indicators, before moving on to ESD as it seems to me that what you can say about indicators applies more narrowly to sustainable development and ESD indicators as well.  In doing this I’ll be drawing on the work that a number of us here did through Anglo-German Foundation grants a few years ago, and also on some more recent sustainable development indicator work in the UK.  First, some words of warning from three people smarter than me:

Where there is no reliable accounting and therefore no competent knowledge of the economic and ecological effects of our lives, we cannot live lives that are economically and ecologically responsible.  Wendell Berry

That which is good and helpful ought to be growing and that which is bad and hindering ought to be diminishing. ... We therefore need, above all else, ... concepts that enable us to choose the right direction of our movement and not merely to measure its speed.  EF Schumacher

We try to measure what we value. We come to value what we measure.   Donella Meadows

I think we all understand this.  The idea and use of indicators – in a general sense – is now widespread, and probably has been for a very long time, and we all experience such indicators everyday.  This shows a range of indicator uses …

Indicators can …

• simplify or distil complex information
• monitor or account for the performance of systems
• measure the state, direction and rate of change
• act as a ‘warning’ system
• raise awareness and communicate information
• stabilise processes – particularly implementation
• aid accountability, governance, decision-making, dialogue
• address stakeholder interests about the state of the system

They find particular uses in quality assurance systems.  Today, we are interested in the idea of indicators in the context of the evaluation of professional practice in the sense of a measure of progress towards a social goal.  Indicators need to be both written with clarity, and systematically developed, and there is a logic to their generation that goes something like this ...

Indicators have to be linked to desired outcomes (which may be short- or longer–term).  They indicate the extent to which outcomes are met

Such outcomes have to be linked to particular purposes (objectives)

Purposes (objectives) have to be linked to overall organisational aims / vision / mission

Key evaluation questions have to be posed, where indicators answer such questions.

To evaluate is to form a judgement about such progress towards a social goal, and indicators help us understand the worth of such judgements.  In our professional practice, the idea and use of key performance indicators is now commonplace.  But this does not make their use easy, or uncontroversial.  Whilst it is important to have indicators, it’s also important not to accept uncritically what they appear to tell us.

The UK government has had 68 sustainable development indicators in place for a long while now, and these are being reviewed this year.  The responsible ministry, Defra, says this:

The Government has committed to measure and report our progress through a new set of sustainable development indicators.  The intention is that these will provide a high level transparent overview as to whether the UK is developing on a sustainable path [and] support our evidence base for policy development across Government.

All very laudable, but I think that this confidence is misplaced.  However, what we’re interested in seems quite similar, in principle, to this.  Looking at the existing sustainable development indicators shows the difficulties.  This is the current UK indicator for education …

The proportion of 19 year-olds with Level 2 qualifications and above

This has the considerable merit of being easy to understand and measure, but what does it really tell us?  Probably, only the number of 19 year-olds with Level 2 qualifications.  This is because the link to sustainable development is tenuous at best, and it’s an act of some faith that this provides any measure that we’re developing on a sustainable pathway.  Maybe the indicator should be the proportion of PhD theses that address sustainable development – except we can recall David Orr telling us that sustainability problems were caused by people with PhDs.   But Orr’s rhetoric is prone to over-simplification.

We have another education indicator in the current UK set.  This one is focused on ESD, and has been under development for about 10 years – and still is.  Is this blank just because we weren’t any good at thinking about it?  Or because it’s impossible to create anything valid and meaningful?   I lean to the latter view, which is why I am so interested in what has been written in this report.  The UK’s new proposals sidestep all these problems quite brilliantly.  There is no mention of ESD, and even education just gets a passing glance.  It’s proposed that we should have headline, and supplementary indicators …

The 12 provisional headline indicators are high-level outcome measures, and capture priority issues for making economic, environmental and social progress … in line with the UK Sustainable Development Strategy’s ‘guiding principles’ of sustainable development:

• living within the planet’s environmental limits
• ensuring a strong, healthy and just society
• achieving a sustainable economy
• promoting good governance; and
• using sound science responsibly.

The headline indicators are:

Economy Society Environment

Economic prosperity                 Healthy life expectancy           Greenhouse gas emissions

Long term unemployment        Social Capital                             Natural resource use

Poverty                                          Social mobility in adulthood    Wildlife & biodiversity

Knowledge & skills                     Housing provision                       Water availability

Those who think these things matter will have noticed that sustainable development has been split into what many see as its component parts.  Sadly, it never gets put back together.  The only reference to education is within the knowledge and skills headline indicator within the “Economy” section.  The documentation says this:

The value of knowledge and skills (as a proxy for human capital) per person of working age), where human capital is defined as “the knowledge, skills, competencies and attributes embodied in individuals that facilitate the creation of personal, social and economic well-being” (OECD, 2001).

Rather ironically, OECD omits any reference to the environmental well-being which is at the heart of the issues we face.  The UK measure is based on gender, age and the level of qualification acquired during participation in compulsory and post-compulsory education, in vocational and tertiary education, etc. None of this is straightforward, and there is no mention of ESD.  And you can see the problem immediately – it’s really just the same as the old indicator – the proportion of the population with level 2 qualifications – though it is somewhat more sophisticated.  This, when taken with the limited view of human capital, means that a lifetime of education and training will count whether or not there is any focus on sustainable development – any ESD if you like.  So, it seems that ESD, if it happens at all, will count for nothing.

However, there’s a much bigger problem.  It has been suggested that such headline indicators represent a sort of barometer or compass bearing.  For example, if all the indicators can be lined up so that they are all pointing in the right direction, then we can conclude that we’re making progress along the sustainable development pathway.  It’s a pleasing, and reassuring, metaphor.  However, there is a major problem with this view.  Reseach that we were involved in, in 2004, noted this:

If the headline indicators are broadly negative, we can tell that the overall position is not sustainable.

Unfortunately, this does not mean that when they are all positive, the position necessarily is sustainable.

Indeed, it is even possible that positive indicator results will operate perversely to move us off a sustainable pathway.

In other words, if you fail to meet an indicator, then you know there’s work to do; if you do meet it, however, there’s always uncertainty about how appropriate the indicator was.  For example, if we were to succeed in keeping the global temperature rise below, say, 3^o C, we’d still have to wait for years to see whether that was enough.  As the old cliché has it: only time will tell.  In this sense, using indicators is rather like looking in a car’s rear-view mirror. This enables us to see where we’ve been – to see how well we’re driving – the sort of social and temporal progress we are making.  On a bad day we can see the mayhem we’ve caused – the crashed cars and bodies in the road.  It alerts us to how we’ve been doing, but is an imperfect guide to the future.  The point about rear-view mirrors is to keep your eye on them – and to try to ensure that they don’t distort what has happened.  These are all important qualities needed in a mirror and an indicator:

accuracy, timeliness, clarity, field of view, ease of access, etc are all crucial in a mirror and indicator and the view you have should be both valid and meaningful

A real danger is that you produce indicators that are easy to measure, but which lack intellectual and policy coherence.

Turning now to ESD, what are the potential benefits and risks of ESD indicators.  Well, they are probably the same as for any other indicator …

Benefit:

You can get (useful) information on steps taken, and progress made, against aims and targets

You can then change policy / practice in the light of information

Risk:

You may be measuring the wrong thing

The information you get may be misleading

It may take considerable time to know whether this is so

So much is straightforward, and applies to ESD as it does to, say, biodiversity.  Both, of course, need a conceptual frame if indicators are to be developed.  Actually, they need this, if goals are to be set.  But, with biodiversity the link to sustainable development seems reasonably clear, whereas, with ESD, this never seems to be the case – at least to me.  This is probably because the conceptual framing of biodiversity is more robust than for ESD.

This is the UK’s proposed new headline indicator measure for wildlife & biodiversity …

Indicator measure:
Wildlife: Bird population indices - farmland birds, (b) woodland birds, (c) seabirds and (d) water and wetland birds

Rationale:
Natural capital includes those elements of the environment that yield resources and ecosystem services, but we cannot determine our entire capital of natural resources and instead have to focus on selected aspects of the natural environment and changes in its state.

The rationale here is that populations of key species of birds are a good indicator of the broad state of wildlife and countryside because they occupy a wide range of habitats and key positions in the food chain.  In addition to this, there are 3 related supplementary indicators that focus on river water quality, fish stocks, the status of species & habitats, and the UK’s biodiversity impacts overseas.

Here, the number of indicators is relatively few, and there is a sound conceptual rationale for the headline, keystone indicator.  Prima facie, this seems good practice.  There is little ambiguity, for example, about bird populations, though there will be methodological issues in the estimation of their numbers.  Of course, these are not actually indicators; rather, they are indicator measures.  The indicator will be how these measures change over time, with upward trends being desired.

But, is the relationship between ESD and sustainable development robust?  On past experience, I’d say not enough, and the difference between practitioners on this matter can be considerable.  Here are two frameworks that have been put forward in the UK higher education context to delineate how we might both think about ESD, and research its effectiveness:

ESD is …

Teaching which develops in students an understanding of the nature of society and its relationship with the environment, together with the capabilities and potential to promote justice in the distribution of economic, social and environmental assets, now and for the future.

ESD is …

Teaching which contains a significant element of work related to either or both of the natural environment and natural resources, plus a significant element of work related to either economic or social issues (or to both).

The first sees sustainable development primarily in terms of increasing equity and global social justice whereas the second sees it more in terms of reducing the environmentally unsustainable impacts of economic and social development.  Are both these ESD?  Can both be ESD?  If so, are these poles on a continuum of ideas, or mutually inconsistent views?  Either way, it’s not useful to try to arbitrate between them.  Except to say, of course, that any attempt to generate indicators for these approaches would lead to quite different outcomes.

Looking at past practice, a lot of the effort that has gone into such indicators has been at the system level and has been about structures and inputs, and quality, rather than about outputs – that is, about what people know, can do, etc.  There has also been the problem of saturation – that is, having an indicator for all occasions.  UNECE couldn’t resist this temptation, trying to cover, with its 50 plus indicators, just about everything.  And there has been what I call the substitution problem.  Here are a number of attributes that might characterise an educational institution that took sustainability seriously, and which could be precursors of indicators.

This is an organisation where issues around sustainability …

• have a high profile across the work of the institution, and in its community links
• are fundamental to, and integral across, the institution’s work rather than being add-on or fragmented
• are raised in different settings, as appropriate, and treated as holistically as possible
• represent one of the institution’s key ethical stances
• are focused on building students’ capability for critical and independent thinking for the future

These are somewhat generic attributes in that they are not obviously about ‘sustainability’, and other ideas could be substituted: health, for example, faith, or enterprise are all possibilities.  This is not an isolated problem.  I remember that my admiration for the logical consistency of an early UNECE indicator document was tempered by the fact that if you replaced ESD by health education (and sustainable development by, say, well-being – or even road safety) it still made remarkable sense.  It seems to me that this just won’t do, as validity questions loom large.  The risk in all this is that we have an approach to ESD indicators where they don’t relate to sustainable development at all; rather they relate just to themselves.  As my colleague (and UNECE indicator group member) Paul Vare has noted, many indicators obscure rather than illuminate, and any meaningful signal is lost in the noise of communication.  But, the following attributes about an educational institution that takes sustainability seriously are, I think, rather different.

A social learning community with a systemic view of the world and a heightened sense of place …

• has a growing awareness of its environmental impact (footprint), has a strategy for steadily reducing it, and uses these as foci for learning
• values outdoor, environmental, experiential and exploratory learning as a means of effectively engaging with real-world issues in authentic settings
• is outward-looking, and whose work in embedded not only in its local context (socially, economically, environmentally, and culturally), but which has tangible links to real communities in other parts of the world
• recognises that place is now a global phenomenon that raises moral issues of inter-dependence and shared responsibility, in relation to social and environmental justice
• understands that [1] it can, and should, contribute not just to maximising learning and skills acquisition (its tradition role), but also to enhancing social cohesion, as well as [2] lessening its need for natural resources and its creation of waste, and maximising the efficiency of its buildings

These seem obviously about sustainability, though there is no mention of this – or ESD. Stephen Gough has argued that, because sustainable development necessarily implies a change in the models we use to live by, then any ESD programme worth its salt would challenge these models by asking questions about how we live – and might live.  He said that we should try to ‘catch the change’ – the dynamic in the educational process, and focus on the choices that have been made in relation to change and to learning.  Such choices and change offer the possibilities of being a useful focus for an indicator of ESD.  With ESD, however, the challenges seem considerable, particularly where materials and courses are concerned.  I say this because it’s not just a question of listing or counting, materials or study programmes.  First, there are issues of quality to be taken into account – are the materials and programmes any good?  In a similar way, one has to ask how healthy bird populations are.

More importantly, perhaps, there seem also to be validity issues.  How will we know whether materials or study programmes are actually about sustainability, or, perhaps, merely only about its component parts?  In other words, it doesn’t seem enough to count a study programme just because it says it’s about sustainability or sustainable development, or because it focuses on component parts.  Rather, don't we need greater conceptual coherence around ESD, and more consensus around criteria as to what’s to count?