I've been reading the research report on outdoor learning recently published by the Blagrave Trust, UCL, the IoL, and Giving Evidence: The Existing Evidence-Base about the Effectiveness of Outdoor Learning.
It will not make comfortable reading for anyone who has been banging on recently about how great outdoor learning [ OL ] is – that is, for most of those involved in OL who want to publicise what they do – that is, for most of them. You don't have to read the report for long to realise how threadbare these various claims turn out to be. It's not helped, of course, that outdoor learning is such a fuzzy 'n' floppy idea: outdoor learning turns out to be any learning that accrues from an experience out of doors, as opposed, say, to indoor learning which ... You get the picture.
Thus the Ten Tors expedition, a field trip to a rocky shore, a scout camp, a nature walk, learning maths at a forest school, a visit to an outdoor theatre, painting al fresco, skinny dipping, fox hunting, detectoring, working on the allotment, a walk with the Ramblers, plane spotting, etc, etc can all lead to outdoor learning, and the only thing they all have in common is that they take place outdoors.
What next, you have to wonder? A claim that indoor learning is good for you as well? But nobody ever talks of indoor learning because it makes little sense to do so (as it is so varied in every sense). Why, then, is it sensible to talk of outdoor learning?
I have thought for a while now that outdoor learning lacked conceptual clarity, and that, because of this, some (at least) of the claims it makes for itself were perhaps open to question. This report suggests that there is no perhaps about it.
Here are a few extracts from the summary of findings
1. A sense of the sector as a whole
There is no comprehensive or regular (repeated) survey of the scale of outdoor learning in the UK. There are some studies of specific outdoor learning activities (e.g., of particular types, or in particular parts of the UK). In these, some authors express concern about barriers to delivering outdoor learning and a reduction in outdoor learning.
2. The current research base
Crowdsourcing UK research revealed an enthusiasm for research and sharing of knowledge amongst people who deliver outdoor learning activities. However, some of the material submitted were data or reflections which included named individuals, rather than anonymized research reports. This raises some issues around practitioners’ understanding of research ethics.
There is a growing body of individual studies and systematic reviews about the development and effectiveness of outdoor learning. We found 15 systematic reviews of the effects of outdoor learning. They provide extensive evidence of the effects of outdoor learning. However, the set is somewhat confusing because many of them overlap in terms of the primary studies they include. Moreover, some systematic reviews include other systematic reviews, or are an update of an earlier review. This overlap therefore repeatedly reports the same evidence without necessarily strengthening it.
Distinctions between types of interventions and outcomes employed to categorise studies are not always clear. For instance, ‘healthy lifestyles’ and ‘health and well-being’ were part of the ‘learning and development’ domain, while ‘health behaviour’ and ‘health, physical / mental’ were part of the ‘health’ domain.
Four features of the 58 primary UK studies are striking:
a. They are spread thinly across many populations (types and age groups), interventions, settings and outcomes, such that few topics have been researched more than a handful of times. This leads to our suggestion that the sector collectively identify and prioritise the important unanswered questions, and then focuses its (presumably limited) research resources on those priority questions.
b. The activities and participants on which studies focus may not be where the sector would choose that research should focus. For example, the most common study topics are: adventure or residential activity; 11-14 year olds; and the general population. This leaves very few studies on (and hence little insight about) other age groups, popular activity such as Scouts or Ramblers, or people who are not in employment, education or training (NEET), have disabilities or are post-trauma.
c. That there seems surprisingly little linkage between the outcomes measured by the studies and the agenda of ‘customers’ and funders. The outcomes measured are mainly around ‘character development-type’ outcomes (communication skills, teamwork, self-confidence etc. Very few studies addressed interventions with strong links to core curriculum subjects. There was only one primary study of educational outcomes at Key Stage 1 (5-7 year olds), few of educational outcomes at Key Stages 2, 3 and 4, and none at or beyond Key Stage 5 (sixth form). There is also a mismatch with the interests of employers: ‘employability’ is only measured in relation to offenders but not young people generally. Looking internationally, only six of the 15 systematic reviews looked at educational attainment, and only one addressed employability.
d. Safety is little covered in the systematic reviews and was not measured as an outcome in any of the primary studies. Safety is obviously a major issue in outdoor learning since it can be dangerous: few social interventions can result in broken limbs or fatalities. Even if safety isn’t the primary focus of a study, data could be gathered about safety: this is often how patient safety data and insights are gathered in medical research.
3. Outcomes assessed
This evidence, both in the UK and internationally, and in both primary studies and systematic reviews, is very varied in terms of the populations who are offered outdoor learning, the type of outdoor learning and the outcomes assessed. The categorisation that informed this study captured some interventions and outcomes, but others emerged from the literature. Generally, there is considerable consensus in the general aims of interventions, but little consensus on the outcomes for assessing their effects.
4. The designs of individual evaluations
We compared reports of UK studies in terms of attributes on a scale developed by Project Oracle, which looks at the extent of plans for an intervention and the evidence for it (described further in the document). Using this scale was challenging because the Project Oracle scale was designed for organisations to plan and assess their own interventions and evaluations, rather than to assess research reported elsewhere.
Many UK studies did not reach Level One of the Project Oracle scale, normally because they did not cite or appear to use a Theory of Change (also known as a logic model: an articulation of the inputs, the intended outcomes, how the inputs are meant to produce those outcomes, and assumptions about context, participants or other conditions). Clear theories of change serve a couple of useful purposes: first, they demonstrate that the practitioners understand their intervention; and second, they are invaluable for other practitioners reading the research in estimating whether they will achieve the same outcomes with those interventions in their contexts. To be clear, a practitioner may have a theory of change but not cite it in their research, but (a) citing it in the research is useful and (b) experience from many other social sectors suggests that practitioners may need support to develop or articulate their theories of change. No UK study, or set of studies, featured the more demanding attributes of Levels Four or Five, around the intervention having been replicated in several places.
Implications for practice and policy
The study did not set out to look at implications of the research for practice and policy. Nonetheless, we found:
Almost all outdoor learning interventions have a positive effect. The effect attenuates over time: the effect as measured immediately after the intervention is stronger than in follow-up measures after a few months. This is common for social interventions. However, one meta-analysis found that effects relating to self-control were high and were normally maintained over time.
Evidence for the value of longer interventions. The systematic reviews found that overnight and multi-day activities had a stronger effect than shorter ones. While this is perhaps unsurprising, it does pose a challenge for funders / funding since it obviously forces a trade-off with the number of participants.
Recommendations
For providers of outdoor learning
Outdoor learning organisations can refer to systematic reviews of research about outdoor learning when planning their programmes. Careful reading is required to (a) check the rigour of each review and the studies they include (for instance, did the review include a systematic search and critical appraisal of the studies included?); and (b) check the precise types of programmes, populations and outcomes they studied.
Implications for the outdoor learning sector about developing its research
Because the existing research is spread quite thinly, few questions about effectiveness are yet answered reliably. We therefore recommend that the outdoor learning sector collectively prioritise the various unanswered questions in order to focus its research resources on those which are most important.
We recommend that the outdoor learning sector:
1. Types and volume of activity: Pull together the various data sources on this to give the current picture, and create a system to regularly capture data on the types and volumes of activity.
2. Improve practitioners’ theories of change, enabling practitioners’ to both create and to use them. Theories of change are explained in Box 4: they are invaluable for understanding why an intervention works and hence whether it is likely to work in other contexts, but only few evaluations of UK outdoor learning activity cited them.
3. Convene practitioners, researchers and others to prioritise research topics.
4. Manage the resulting sector-wide research agenda, through relationships with funders, and possibly by creating partnerships between practitioners and researchers.
5. Ensure that both interventions and research are described clearly, fully and publicly.
Outdoor learning organisations need to have systems in place to support ethical practices for monitoring and research, particularly the storage and sharing of data from evaluations.
Greater consensus about the important outcomes of interest would allow research findings from different studies to be pooled more easily, and thereby facilitate accumulating knowledge to inform better the whole field.
.....................................
Enough ...
Respond