Bridging the Gap between Collaborative and Realist Evaluation: A General Critique and Case Study of European Union Employment Policies
Carpenter, Mick, International Journal of Action Research
After first establishing the need for a reflexive approach to evaluation, the article focuses on the theoretical breakthrough of the 'fourth generation' evaluation paradigm proposed by Guba and Lincoln (1989). This is seen as having much to offer, but their stakeholder focused approach has the danger of lapsing into relativism. Therefore a 'realist' approach is proposed building on Pawson and Tilley (1997), going beyond deconstruction of stakeholder perspectives to make verifiable statements about programme and project effects. This is then worked through with an outline review of the issues associated with evaluating European Union Employment Strategies. The article concludes by suggesting that a realist concern for critical praxis in evaluation can also bring it closer to action research paradigms.
Keywords: Evaluation, realism, employment, European Union
The purpose of this paper is sympathetically to analyse emerging critical traditions in evaluation and find ways of bridging collaborative and 'realist' traditions, working through the concrete example of European employment programmes. I argue that while evaluation should always start with the perceptions and understandings of stakeholders, the evaluator necessarily has an independent role in assessing which have greater plausibility rather than simply stop at synthesising different accounts. This is because there is the need to allow for the possibility of contradictory or irreconcilable meanings, and also because 'deeper' causal processes may lie at a level not easily accessible to local actors. Thus while evaluation rightly has a pragmatic concern with improving programmes and/or organisations, and helping stakeholders listen and respond to each other, it should not abandon a fundamental concern with uncovering causal processes. 'Realist' approaches, which incorporate social meanings and processes into the causal analysis of change, can help to provide a 'middle way' between extremes of social constructionism and positivism. While the elitist tendencies of objectivist evaluations might be acknowledged, and a preference for participative evaluation from below asserted, there is no need to throw out the scientific baby with the positivist bathwater. In other words, while I sympathise with collaborative evaluation's efforts to bridge gaps between internal actors and external judges, evaluators bring something to different to the table and need to be more than non-judgemental project conciliators.
These reflections arise from evaluative research funded from the European Social Fund (ESF) EQUAL programme undertaken as part of the Great Britain wide SEQUAL project aimed at identifying good practice in integrating excluded and discriminated groups into the labour market. It also arises from discussions held within the evaluation sub-group of SEQUAL's transnational partners in FACETS, which sought to develop cross-national learning about labour market integration of migrants and ethnic minorities (for information on SEQUAL and FACETS see http://www.surrey.ac.uk/politics/ cse/sequal.htm). The first part of the article seeks to address general issues, and these are then applied in outline to the specific context of the European Union's employment policies.
2. Deconstruction and beyond
At a time when a pragmatic and rather mechanical concern with 'what works' has increasingly come to dominate public policy, seeking to utilise evaluation as a set of depoliticised technologies for evidence based public policy (see for example, Davies/Nutley/Smith 2000), it is necessary to assert explicitly the political and critical nature of evaluation as a reflexive activity (Taylor/Balloch 2005). The advantage of the constructivist tradition to evaluation is that it brings the political nature of evaluation to the fore and leads to a critical questioning of the way that ideological purposes are incorporated within evaluation strategies. …