Academic journal article Australian Health Review

Evolution of a Multilevel Framework for Health Program Evaluation

Academic journal article Australian Health Review

Evolution of a Multilevel Framework for Health Program Evaluation

Article excerpt


Evaluating health programs presents many challenges, arising from factors such as the social world within which programs are implemented, the complex nature of the programs being evaluated and the need for methodological rigour. This necessitates an approach that can encompass stakeholder engagement and investigate the full range of outcomes that may be relevant to those stakeholders. In addition to judgements about worth or merit, judgements about sustainability and the possibility of wider implementation are also typically required. The evaluation approach should frame the language and methods of the evaluation in a way that facilitates understanding and participation from multiple stakeholders.1,2

Herein we describe the development and use ofan evaluation framework by the Centre for Health Service Development (CHSD) at the University of Wollongong. The framework has strong face validity, is easy to understand and has demonstrated applicability across numerous health program evaluations. In the context of using the framework, face validity is framed in terms of reasonableness, in which the question becomes: ?In this situation, given what we know about what the program did, and given what we know about the impact of the program, is it reasonable to assume that there was some significant amount of linkage between what the program did and what resulted?? .3 The framework was developed over a 6-year period while conducting a range of evaluations in collaboration with different providers, program managers and stakeholders. The current version of the framework has been in use since 2003.

Development of the framework

The genesis of the framework was the evaluation of the Illawarra Coordinated Care Trial from 1997 to 2000, part of an Australiawide series of demonstration projects to assess the benefits of coordinated care for older people living at home who either had a risk of falling or complex medical or social problems requiring services from multiple health service providers.4 The evaluation was informed by research that identified eight factors that affect whether a context is receptive to change.5 These factors were used as a way of organising the evaluation and summarising the conclusions in a series of reports that described impacts and outcomes for clients, providers and 'the system'.6

The lessons from that evaluation were used to guide the evaluation of demonstration projects that sought to test ways of integrating private psychiatrist services and public sector mental health services, structured around an evaluation hierarchy consisting of three levels:7

· Level 1: consumers

· Level 2: providers seeing patients, in both the public and private sectors

· Level 3: the system as a whole.

Further development of the evaluation framework occurred with the evaluation of a new model for delivering palliative care in Griffith in New South Wales that incorporated the concepts of capacity building, sustainability and generalisability.8,9 Conceptualisation of capacity building and sustainability was based on a study that had developed a set of indicators of capacity building in health promotion for organisations, groups and communities.10,11 The study sought to capture what was described as the 'invisible side' of health promotion (i.e. capacity building). Development of the indicators was informed by a review of the literature on capacity building.12

Subsequently, the framework was refined to evaluate two national palliative care programs: the Caring Communities Program13 and the Rural Palliative Care Program.14 The basic structure of the three 'levels' was retained, but with refinements to the definition of each level:

· Consumers: patients, families, carers, friends, communities

· Providers: professionals, volunteers, organisations

· Care delivery system: structures and processes, networks, relationships.

Input regarding the framework was obtained from an expert panel and National Reference Group established for the Caring Communities Program. …

Search by... Author
Show... All Results Primary Sources Peer-reviewed


An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.