{"id":880,"date":"2017-12-05T08:46:53","date_gmt":"2017-12-05T08:46:53","guid":{"rendered":"http:\/\/blogs.bath.ac.uk\/iprblog\/?p=880"},"modified":"2017-12-05T10:15:39","modified_gmt":"2017-12-05T10:15:39","slug":"preventing-violent-extremism-the-problem-of-meaningful-evaluation","status":"publish","type":"post","link":"https:\/\/blogs.bath.ac.uk\/iprblog\/2017\/12\/05\/preventing-violent-extremism-the-problem-of-meaningful-evaluation\/","title":{"rendered":"Preventing violent extremism: the problem of meaningful evaluation\u00a0"},"content":{"rendered":"

Giorgia Iacopini is\u00a0Senior Researcher and Consultant at the Tavistock Institute of Human Relations<\/a> in London, and is studying on the IPR's Professional Doctorate in Policy Research and Practice<\/a> (DPRP).\u00a0<\/em>This blog is based on a conference paper that was accepted for the BISA Critical Studies on Terrorism working group<\/a>'s 2017 annual conference.<\/em><\/p>\n

 <\/p>\n

Preventing Violent Extremism (Prevent) is part of the UK\u2019s counter-terrorism policy (CONTEST), established in 2006 in response to the London Bombings in 2005. As the name suggests, the aim of Prevent is to stop people from being drawn into terrorism and therefore operates in what is commonly referred to as the \u2018pre-criminal\u2019 space. Central to the policy\u2019s theoretical foundation is the notion of \u2018radicalisation\u2019, defined as the process through which individuals are potentially drawn into violence[1]<\/a>.<\/p>\n

Prevent can be seen as part of a much wider effort, in Europe and globally, to address what has become an important global security concern.\u00a0 Indeed, over the past decade there has been an exponential growth of what are defined \u201ccountering violent extremism\u201d (CVE) policies, programmes and initiatives which, like Prevent, are specifically aimed at deterring individuals from radicalising to violence.<\/p>\n

But is all of this working?<\/strong><\/p>\n

Despite the decade-long investment, answering this question is still not straightforward. Indeed, policy makers, practitioners, and researchers across Europe (and worldwide) have increased calls for identifying 'what works' in interventions that seek to counter violent extremism. This has resulted in growing volumes of reviews and debates on the \u2018state of the art\u2019 of CVE evaluations, as well as efforts to pull together examples of possible metrics to identify \u2018success\u2019 and guidance on the evaluation approaches available[2]<\/a>.<\/p>\n

On the whole, however, one of the main criticisms made against policies and programmes in this field is that they lack empirical data around their effectiveness. One could argue that the considerable evaluation challenges in this area of work have been a constant feature. Why is this the case?<\/p>\n

First, evaluating preventive efforts against radicalisation or violent extremism runs into the methodological challenge that is frequently encountered in evaluations that are required to establish \u2018what works\u2019, which relates to the extent to which it is possible to identify causal linkages in a field where there are many intervening variables (complexity<\/em>). This is the so-called \u2018attribution dilemma\u2019 (ie. being able to prove that the intervention alone was responsible for an observed change). In turn, demonstrating impacts in this field requires, ideally, studies that track changes (in attitudes, or behaviours, for example) associated with the projects over a significant period of time. In the UK at least, this is hindered by commissioning cycles, which see local projects not funded long enough for outcomes to emerge \u2013 a common obstacle for many third-sector organisations[3]<\/a>. In the case of Prevent, the problem is compounded by yearly funding streams (rather than multi-year funding), creating a situation in which there is a pressure to demonstrate results in a very short period of time. The implication is that it becomes almost impossible to explore potential changes that may take many years to materialise.<\/p>\n

A third set of challenges is represented by the fact that Prevent work is situated in a policy and practice field that is continually evolving (evolution<\/em>). Policy directions and innovations are constantly refined and updated, and theory and practices around the causes of violent extremism and what can be done to promote awareness-raising and\/or behaviour change continually evolve.<\/p>\n

Finally, there is the challenge related to variety of local delivery (context-specificity<\/em>). To put it bluntly, local context matters a great deal for success. To have any chance of succeeding, providers have to design interventions that fit the local need and are locally-relevant, which requires consideration of the cultural, social characteristics of the area. The implication here is that Prevent delivery requires creative solutions; identifying \u2018one size fits all\u2019 approaches or developing common measurements of success that bypass local specificity is potentially inappropriate.<\/p>\n

To sum up, therefore, the core evaluation challenges relate to: the complexity<\/em> of Prevent, which makes attribution difficult \u2013 perhaps impossible \u2013 to establish; the evolution<\/em> that characterises the policy and practice field; and the context-specific<\/em> nature of its implementation. By extension, evaluating Prevent would greatly benefit from evaluation approaches that can: overcome the problem of attribution; capture the complex and sophisticated nature of Prevent implementation; and take into account local variation.<\/p>\n

In Prevent, however, the evaluation \u2018trend\u2019 has gone in a different direction, albeit in an effort to strengthen \u2013 rather than hinder \u2013 it[4]<\/a>. The conceptual, organisational and operational shifts in the policy which occurred in 2009, 2010 and 2015[5]<\/a>\u00a0can perhaps explain the challenges of its evaluation.<\/p>\n

The first set of shifts saw Prevent move (and widen in scope) from a focus on targeting violent extremism to also targeting non-violent extremism, raising the importance of \u2018ideology\u2019 as a key \u2018root cause\u2019 that can draw a person into violence. This also included a focus on identifying \u2018individuals at risk\u2019, partly done to overcome concern that local Prevent delivery had not been focused or \u2018hard edged\u2019 enough (hence not addressing the challenge). As a result, the second shift related to the policy\u2019s ownership: Prevent went from being a joint endeavour between the Department for Communities and Local Government (DCLG) to being the responsibility of the Office for Security and Counterterrorism (OSCT). This resulted in increased centralisation (with the OSCT being in charge of policy direction and funding) and reduced funding for projects that were not deemed in line with Prevent\u2019s aims. The third, and most recent, change was the establishment of the Prevent duty in 2015, which for the first time required public sector institutions (local authorities, schools, colleges, universities, health services, prisons and probation) to formally assess the risk of radicalisation.<\/p>\n

These changes guided Prevent\u2019s monitoring and evaluation approach. First, the rise of \u2018indicators of risk\u2019 as a way of assessing success translated into a preference for evaluation approaches that were positivistic in nature \u2013 ie. (quasi) experimental designs. This is very much in line with the rise of \u2018evidence-based\u2019 policymaking which is rooted in a largely quantitative\/positivistic paradigm, as illustrated by the notion of the \u2018hierarchy of evidence\u2019 which places experimental designs at its top[6]<\/a>. The idea is that this evidence-based knowledge can provide clear and confident answers about \u2018what works\u2019, offering a blueprint able to assist politicians and policymakers in their decisions about which interventions to fund, thus avoiding wasting public resources on those that do not work. Second, the move towards greater centralisation, and the therefore reduced local autonomy to design interventions, was perhaps an attempt to \u2018solve\u2019 the problem of local variation in project delivery. \u00a0Third, the Prevent duty was a way of collecting consistent data on the \u201cnumber of individuals within interventions programmes or total expenditure on Prevent in a sector\u201d<\/em> (input) and \u201cthe number of individuals no longer assessed as being vulnerable or a reduction of risk individuals no longer assessed as being vulnerable\u201d<\/em> (impact)[7]<\/a>.<\/p>\n

From an evaluation point of view, this is understandable, and perhaps even seductive. Indeed, reducing local variation would enable more straightforward\u200b\u200b\u00a0comparisons between interventions. This would facilitate the task of understanding of what appears to work best, generating a pool of knowledge able to inform decisions about which interventions to replicate.<\/p>\n

But is this really the case?<\/strong><\/p>\n

Arguably not. Reducing the complexity of the policy, or the nature of the challenge or the opportunity for local variation also reduces the opportunity to evaluate meaningfully. Prevent displays characteristics of what are now being commonly defined as complex interventions. These are: nonlinearity and unpredictability of change processes (ie. the factors that lead people to violence are extremely complex and varied, and it is difficult to isolate how a single programme affects the many factors that contribute to a person\u2019s involvement in violent extremism); local adaptation of projects, which speaks to the importance of context; multi-actor delivery; and multiple interventions being delivered at the same time and that therefore influence each other, making \u2018attribution\u2019 difficult to ascertain[8]<\/a>. Individual programmes can therefore only be evaluated meaningfully against indicators that are context-specific. This runs counter to the standardisation drive that has characterised Prevent since 2011 and even more so from 2015.<\/p>\n

The perhaps more significant question that we may need to ask ourselves is the extent to which the demand for a certain kind of evaluation would be seen as valid by those whose task it is to translate the policy into practice. The more prescribed approach to Prevent following the more recent changes in the policy and the subsequent reduction of local autonomy has not removed the need, locally, to \u2018fit\u2019 the project in the community. Standardising approaches to delivery and to data collection tools therefore may not provide an adequate understanding of project implementation.<\/p>\n

This speaks to one of the biggest challenges in evaluation, which relates to the need to bring together \u2018scientific\u2019 and \u2018stakeholder\u2019 credibility[9]<\/a>. The former relates to the generating evidence that will stand up to scrutiny. The latter refers to the extent to which those affected by the evaluation have a say in its design, or see its relevance. Without stakeholder credibility, the ability to use evaluation findings is absent. The difficulty stems from the fact that these two aspects may be at opposite ends: scientific rigour (ie. as close as possible to the aforementioned \u2018gold standard\u2019) might either be impractical to fulfil or may not be seen as useful to stakeholders due to its \u2018reductionist\u2019 philosophy that clashes with their (often more complex) views on social, community problems and how to solve them.<\/p>\n

Seeking causality between an intervention and observed results is of course a crucial part of evaluation and (quasi) experimental approaches are undoubtedly important for this endeavour. However, they tend to have linear notions of how change happens (A causes B). While this works well where there are known solutions to problems and stable contexts, it is problematic for interventions that that do not have these attributes. Should we therefore be making make Prevent evaluation explicitly complexity-consistent? This would mean designing an evaluation that supports ongoing learning and reflection; surfaces the diverse assumptions about how and why changes occurs; and explores initiatives\u2019 progress and impact through evaluation findings that are grounded in local contexts[10]<\/a>.<\/p>\n

 <\/p>\n

 <\/p>\n

Footnotes<\/strong><\/p>\n

[1]<\/a> HM Government, 2011b. Prevent strategy 2011 - GOV.UK<\/em>.<\/p>\n

[2]<\/a> For example:<\/p>\n