Assessment of Professional Development for Teachers in the Vocational Education and Training Sector: An Examination of the Concerns Based Adoption Model
Saunders, Rebecca, Australian Journal of Education
Professional development has long been viewed as a capacity-building mechanism for teachers, and is widely accepted by governments and organisations as a means to leverage change (Corcoran, 1995; Corcoran, Shields & Zucker, 1998; Fullan, 2001; Guskey, 2002; Lieberman & Pointer Mace, 2008). What constitutes effective Provision is a site of contention (Garet, Porter, Desimore, Birman & Suk Yoon, 2001; Guskey, 2002) and--in the vocational education and training (VET) context discourses surrounding teacher professional development--tends to focus on profiling the competencies required to operate effectively in VET systems negotiating escalating rates of cultural, political, economic and technological change. Amongst the most notable contributors in this debate are Attwell (1997); Cort, Harkonen and Volmari (2004); Dickie and colleagues (2004); Guthrie (2010); Guthrie and Clayton (2010); Harris, Simons and Clayton (2005); Mahlamaki-Kultanen, Susimestsa and Ilsley (2006); Mitchell, Chappell, Bateman and Roy (2006); Mitchell and Ward (2010) and Volmari, Helakorpi and Frimodt (2009).
While there is some discussion in the literature about the design, implementation and assessment of VET teacher professional development (for example, Australian National Training Authority, 1997; Guthrie, 2010; Guthrie, Perkins & Nguyen, 2006; Loveder, 2005; Wheelahan & Moodie, 2010), the scholarly literature base on empirical assessment and evaluation of programs using validated and reliable measures still remains small (Mahlamaki-Kultanen et al., 2006). Despite this, there is no shortage of reports about professional development activities in VET. Some of the biggest contributors to this body of work are from long-term national professional development programs, such as Reframing the Future, which ran from 1998 to 2008, and the e-learning professional support program called the Australian Flexible Learning Framework (2000--2012). Both programs have received significant funding over the years, supported hundreds of projects and have been viewed as key enablers of supporting professional development and change in VET. But research suggests that there are improvements to be made in the assessment and evaluation of such programs. An evaluation of the Reframing the Future program in 2004 found that reporting and evaluation processes relied heavily on self-report data that 'focused on project activities, processes and reactions of participants rather than outcomes' (Caven, 2004, p. 10). Recommendations in the report call for 'an appropriate measurement system' (Caven, 2004, p. 10) that includes quantifiable indicators. Evaluations of the Australian Flexible Learning Framework have focused predominantly on the broad uptake and use of e-learning, and a report on systemic change initiatives in Australian VET published by the Organisation for Economic Co-operation and Development (OECD) found that, while impressive, the evaluation mechanisms relied heavily on self-report data from participants, and independent criteria and measures were not used.
In this study, the Concerns Based Adoption Model is described as a conceptual lens and methodology for the assessment of professional development programs. The Concerns Based Adoption Model is not presented here as a panacea for assessing VET professional development but instead its applicability and usefulness for program assessment are described and discussed in relation to a four-year system-wide professional development program for VET teachers in Western Australia. Since its development in the early 1980s, this model has been widely Used for measuring and explaining educational change, including that resulting from professional development, and it is arguably one of the more conceptually robust and empirically grounded models for examining change (Anderson, 1997). Drawing initially from the work of Francis Fuller (1969), the Concerns Based Adoption Model was developed by researchers at the University of Texas at Austin in the Late 1970s and early 1980s. The model is built on the premise that change is a process, not an event, and as individuals navigate their way through the process they encounter not only a number of affective 'stages of concern' but they also progress through different 'levels of use' (Hall & Hord, 1987, 2006; Hall & Loucks, 1979; Loucks, Newlove & Hall, 1975).
While the Concerns Based Adoption Model has been used extensively within the US K--12 school environment, to date there has been little application of the model within VET. In this mixed methods study, findings from the first two phases of a larger research project are used to profile the journeys of professional change experienced by 27 VET teachers in Western Australia. These teachers participated in a four-year systemic change professional development initiative designed to extend their instructional practices. While this study is Australian, the usefulness and benefits of the approach are discussed in terms of their implications for the design, implementation and assessment of VET professional development initiatives on a wider scale.
Vocational education and training
VET performs an important function in preparing individuals for work and life in the 21st century. In most OECD countries, VET is charged with helping maintain the competitiveness of industries and national economies (Rauner & Maclean, 2008) and is closely linked to national human capital and workforce development agendas (European Centre for the Development ofVocational Training, 2009, 2010; Loveder, 2005). Maintaining a productive VET system capable of operating within a complex and changing environment therefore depends critically on continuing to develop the knowledge and skills of those at the heart of the system--VET teachers (Attwell, 1997; Cort et al., 2004; Volmari et al., 2009). As VET teachers attempt to come to terms with the growing demands placed on them by governments, industries and learners, it is not surprising that debate has intensified regarding how best to deal with the workforce development needs of this group. There have been a number of studies exploring the diversification of roles and changing work practices of VET teachers. A recent study by the European Centre for the Development of Vocational Training (CEDEFOP) also revealed, 'serious Discrepancies between the training of professionals in VET and their work.' (Volmari et al., 2009, p. 3). In the current climate these issues give rise to important questions and have significant implications for the nature, design, implementation and assessment of professional development for VET teachers.
Vocational and educational training in Australia
Over the past 10 years Australia has benefited from a booming economy, closely Tied to growth in Asia. Economic expansion has occurred at such an unprecedented rate that the country recently found itself under-prepared to meet the challenges of significant skills shortages in defined occupations. In response, state and federal governments implemented a number of policy changes. At the front-line of enacting these reforms are VET teachers, who are being called upon to deliver more skills to more people across diverse markets that are influenced by fluctuations in industry and the economy. These changes demand that VET teachers change their views and approaches to training and develop new skill sets and ways of operating (Cort et al., 2004; Harris et al. 2005; Moynagh & Worsley, 2003).
In an attempt to adopt a more proactive approach to dealing with the impact of global challenges, the ministerial advisory body Skills Australia presented its National Workforce Development Strategy to the Australian government in early 2010. This document outlined a range of strategies designed to sustain national economic growth and increase productivity (Skills Australia, 2010). In recognition of the fundamental role VET and those who work within it have to play in realizing this vision, Skills Australia recommended that the government invest $240 million over the next six years to build the capacity of the Australian tertiary education system. The release of Skills for prosperity--a roadmap for vocational education and training (Skills Australia, 2011) further strengthened this position. Outlining major reforms for the sector, including a commitment to invest $1.75 billion over five years in VET, the report highlighted the strategic positioning of the Australian VET sector as a 'principal instrument' (Skills Australia, 2011, p. 1) in driving Economic and demographic change.
Professional development for VET teachers
Professional development provision for VET teachers will undoubtedly play an integral role in facilitating this national change agenda, but the design, implementation and evaluation of such professional development are open for debate. In the Australian VET sector, professional development activities are predominantly designed to meet priority skill needs, often linked to implementing aspects of the national training system and responding to system compliance issues (Schofield & McDonald, 2004). Wheelahan and Moodie (2010, p. 49) found that the majority of programs in VET are 'event focused', rolled out as 'just in time'; they are designed to meet the latest VET policy revision (Guthrie & Clayton, 2010; Harris et al., 2001; Perkins, 1997). Underpinning the design of these programs are assumptions that there is a gap in skills or knowledge, that new information is given, that learning occurs and that change in practice results. Funding models also largely reflect this short-term perspective, providing seed funding that fails to support any long-term systemic embedding of new skills and knowledge (Harris et al., 2001; OECD, 2008, 2009). A significant body of research supports the view that short-term approaches focused on promoting the latest political initiative work against building emergent practices; this is a critical design flaw when attempting to initiate and embed longterm sustainable change (Cort et al., 2004; Dickie et al., 2004; Forewood, Mclean & Butler, 2001; Guthrie, 2010; Guthrie & Clayton, 2010; Harris et al., 2001; Villegas-Reimers, 2003; Wheelahan & Moodie, 2010; Wilson, 2003).
Between 2002 and 2008, $1.5 million per annum was spent in Western Australia on professional development for VET teachers through the Teaching Learning and Assessment Strategy Group program. It was under this program that the professional development program described in this article was funded. A review of these activities in 2006 recommended that they needed to be more strategically positioned, and robust evaluation mechanisms were required to determine the programs' impact (Perkins, Guthrie & Nguyen, 2006). This view is consistent with other studies, already outlined, that draw attention to the lack of systematic research and assessment of the impact of change and professional development initiatives in Australian VET and call for greater accountability by increasing the systematic assessment of publicly funded programs (Guthrie, 2010; McDonald, 1999; OECD, 2008, 2009). A recent report on the VET workforce produced by the Productivity Commission (2011) found that data on professional development expenditure was 'patchy' (p. 277) and that the sector lacked an evidence base to inform decision-making about professional development. The report also supported calls from the Victorian TAFE Association and the TAFE Directors Association for the use Metrics to measure and capture 'meaningful data' (Productivity Commission, p. 280) about the effects of professional development, which in turn can be used to inform future planning and implementation. This position is further supported by the findings from a study conducted by the OECD on VET systemic reform projects, which identified the Australian VET sector as having a 'weak evaluation culture' (OECD,2009, p. 52) with no frameworks or feedback loops to help establish evidence-informed approaches for the future design and implementation change initiatives. The report called for an increased focus on evaluation that is based on evidence and research and also for the establishment of guidelines for measuring efficacy and increasing accountability (OECD, 2009).
Educational change and professional development
There is a substantial body of research on educational change and its relationship to teachers' professional development (Fullan, 2001; Fullan & Hargreaves, 1992; Hall & Loucks, 1979; Hord, Rutherford, Huling & Hall, 2004; Huberman, 1983; Huberman & Miles, 1984; Joyce & Showers, 1995; Joyce & Weil, 1996). The role of professional development as a mechanism for supporting the implementation of change is widely accepted and a summary of research reports into professional development by the Australian National Training Authority (1997) identified it as a key enabler of supporting the change process in the VET sector. Although theories of change often underpin the design and implementation of models of professional development, like that of the Instructional Intelligence program, often too little attention is paid to the research and to what constitutes effective professional development (Bennett & Rolheiser, 2006; Fullan, 2001). But the research corpus is clear and broadly agrees on several features of change:
* change is a process, not an event
* change is made by individuals first, then by organisations
* change is a personal experience and evokes emotional and behavioural responses based on individual thoughts and feelings
* change takes time.
In addition Fullan (2001) has identified three dimensions to change:
* possible use of new or revised materials--this can include curriculum materials and instructional resources
* possible use of new teaching approaches, strategies or methodologies
* possible change in beliefs--challenging the existing assumptions or theories that underlie programs or policies.
More often than not a change initiative is concerned with all three of Fullan's dimensions, as each is intrinsically linked in teaching and learning practice. An understanding of the developmental nature of change is vital when implementing professional development and designing models for it, as the models need to embody strategies that acknowledge that change is a process that occurs over time. Fullan (2001) identified three broad phases of this process:
* Phase I--initiation (the process leading up to and the decision to proceed with change)
* Phase II--implementation (putting into practice new ideas and practices)
* Phase III--continuation (sustaining the program and its effects after implementation)
Depending upon their level of involvement, individuals may not experience all stages of the process and often will not be at the same stage in the process as their colleagues. Time phases are not clearly demarcated and are often re-negotiated; moderately complex change can take approximately three to four years to complete and significant change can take up to 10 years. So what robust and reliable evaluation methods that acknowledge change as a dynamic and complex process can inform our conceptual understanding, how can we use them and of what value are they to us?
The purpose of this article is to examine the usefulness of the Concerns Based Adoption Model in the context of state systemic reform in the Australian VET sector. Specifically, the following questions related to professional development provision for VET teachers are explored:
* What information does a Concerns Based Adoption Model provide? How does this facilitate a better understanding of VET teachers' responses to professional development initiatives requiring substantial change in instructional thinking and practice?
* In what ways can the Concerns Based Adoption Model be used for professional development program assessment in the Australian VET sector?
There are numerous models and approaches to planned change (Cummings & Worley, 2008; Fullan, 2001; Kotter, 1996; Senge, 1990) and a growing body of literature in VET that also speaks to 'bottom up' or localised change in the form of innovation (Figgis, 2009; Figgis & Hillier, 2009; Smith, Courvisanos, Tuck & McEachern, 2012). Similarly, there are many approaches to evaluating or Assessing the impact of change initiatives: for example, Kirkpatrick's levels of learning evaluation model (Kirkpatrick & Kirkpatrick, 2006) and Unger and Rutter's (1997) strategic training evaluation model have both received attention and application within the VET context. While these models and approaches are valuable, the Concerns Based Adoption Model is the specific focus of this article for a number of reasons:
* it is specifically designed to measure the impact of reform programs or initiatives for educators (Hall & Hord, 1987, 2006; Hall & Loucks, 1979; Loucks, Newlove & Hall, 1975). Based on systemic change theory, it recognises educational change is multifaceted and involves the complex and dynamic interplay between people, organisations, systems and processes.
* it comprises a researched conceptual framework for change, proposes principles for effective change implementation and profiles the styles of change agents. The model is composed of three dimensions that act as lenses through which to view the change process and these potentially afford insight into how change is experienced and mediated at individual and group levels. The three dimensions are 'stages of concern', which relates to the affective experience of change, 'levels of use', which focuses on the behavioural dimension, and 'innovation configurations', which is used to profile the different ways in which the change process should be implemented.
* each dimension of the framework has a corresponding set of methods specifically designed to measure individuals' feelings, perceptions and behaviour in response to change initiatives, including professional development.
Not only does the Concerns Based Adoption Model provide a framework for guiding the design, construction and implementation of professional development programs across a diverse range of educational and training systems and settings, it also provides a set of researched valid and reliable measures that can be used to assess the impact of change initiatives (Hall & Hord, 2006; Hord et al., 2004).
The Instructional Intelligence professional development program
The professional development program that provided the vehicle for this study is known as Instructional Intelligence (Bennett, 2002, 2010). The program was initiated in response to a change in West Australian state legislation that permitted senior secondary school students (15--17 years old) to enter the VET system; it aimed to supportVET teachers to engage this cohort by extending their instructional repertoire and ran for four years (2005--2008).
The model of professional development used to implement the Instructional Intelligence program was based on research into educational change and staff development, recognising that change can take time and that effective staff development occurs when individuals work in teams, have opportunities to practise, reflect on progress and receive constructive feedback and coaching (Fullan, 2001; Joyce & Showers, 1995; Showers, Joyce & Bennett, 1987). All program participants were volunteers and attended workshops in teams of between two and four. Workshops were held twice a year; each workshop session ran for three days. At each session the program consultant modelled a selected range of instructional innovations, and participants considered the theory and research that supported the use of these innovations. For example, participants learnt how to use graphic organisers such as concept maps, Venn diagrams and mind maps to present content and assess student work. They considered the potential impact and process of integrating innovations across different content domains and with different cohorts of students. Participants returned to their colleges to integrate the new instructional methods into their practice and reflected on their progress at regular team meetings.
In Western Australia, there are 11 publicly funded VET colleges, comprising 50 campuses situated in metropolitan, regional and remote locations. They offer over 400 full-time courses that lead to nationally and internationally recognised vocational qualifications issued under the Australian Qualifications Framework. All research participants in this study worked in the public VET system in Western Australia and were recruited from the group of 35 VET teachers in the Instructional Intelligence professional development program. A total of 27 teachers Volunteered to take part in this research. This group is broadly representative of VET teachers in Western Australia, working across diverse content and vocational areas including: adult literacy; business studies; graphic design; metal, mining and engineering trades; building and construction; and community services.
Research participants comprised 8 male and 19 female teachers and were distributed across the 11 college campuses in metropolitan, regional and remote locations (see Table 1). Fourteen participants were from regional colleges, four were from remote locations and nine were from metropolitan colleges. Given there are four metropolitan and seven regional and remote colleges within the West Australian public system, the sample is approximately evenly spread across the geographical distribution of colleges.
Participants varied in their teaching experience and the number of years they Had participated in the program. The majority of the sample could be described as experienced, with 17 having 11 years or more teaching experience; six of this number had been teaching for more than 20 years. Out of the remaining 10 participants, two had been teaching for between one and four years and eight for between five and ten years. Seventeen out of the 27 had participated in all four years of the program, four for three years and six for two years. (Relationships between the amount of time participants had been involved in the program and their Concerns Based Adoption Model profiles are discussed later.)
The Concerns Based Adoption Model
As noted previously, the Concerns Based Adoption Model comprises three dimensions that act as lenses through which to understand the change process and gain insight into how change is experienced by individuals and groups:
* stages of concern, which focuses on the affective domain (how individuals feel about the process)
* levels of use, which assesses the behavioural aspects of change (ways in which individuals put professional learning into practice)
* innovation configurations, which describe the various forms of an innovation that educators adopt during implementation.
For this study, the Concerns Based Adoption Model provided an appropriate model to conceptualise and assess the impact of a systemic change. For this study, the 'stages of concern' and 'levels of use' dimensions were used as they provide insights into change at the individual level: how it is experienced and felt, and the extent to which new practices and processes are implemented.
The stages of concern dimension captures individuals' feelings, preoccupations and perceptions towards implementing change in their context. There are several assumptions:
* change is a developmental process that occurs over time
* as individuals progress through change they experience a range of feelings
* although there are seven defined stages of concern grouped into self, task and impact not all individuals follow this progression at the same rate or in a linear fashion
* it is typical for individuals to display a combination of concerns simultaneously (George, Hall and Stiegelbauer, 2006; Hall & Hord, 2006).
The seven stages of concern are detailed in Table 2.
The Stages of Concern Questionnaire quantifies these seven different stages of concern. The instrument comprises 35 Likert scale questions in which respondents are asked how they feel about the professional change that they are experiencing. The following scale is used with all items: (0) irrelevant, (1-2) not true of me now, (2-4) somewhat true of me now and (5-7) very true of me now. Hall, George & Rutherford (1979) determined that the Stages of Concern Questionnaire has good reliability, with Cronbach-a (internal consistency) coefficients ranging from 0.64 to 0.83, and the test-retest (Pearson-r) coefficients ranging from 0.65 to 0.86 (Hall & Hord, 2006). Hall, George and Rutherford (1998) used a number of approaches to determine the validity of Stages of Concern Questionnaire scores including inter-correlation matrices, judgements of concerns based on interview data and confirmation of anticipated group differences and changes over time. The results of these studies indicate that the reliability and validity of the Questionnaire are within acceptable ranges for assessing individual concerns within Professional change initiatives.
The levels of use dimension of the Concerns Based Adoption Model considers the behavioural aspect of change; specifically, to the extent to which individuals are implementing new practices. This dimension is founded on assumptions similar to those of the stages of concern: that change is a developmental process, that individuals move through defined levels of use and that they are unlikely to all move at the same rate or in a linear fashion (Hall, Dirksen & George, 2006; Hall & Hord, 2006). Levels of use are detailed in Table 3.
Individual levels of use can be determined using either the Levels of Use Branching Interview or the Levels of Use Focused Interview. For both protocols, an individual's placement at a level of use is determined by decision points (Hall & Hord, 2006). For the branching interview the interviewer conducts a short, informal interview to gain a broad view of an individual's level of use. The focused interview is a longer, more formal process and was selected for this study because it affords deeper probing into the implementation of the innovation. This interview takes 30 to 40 minutes and involves asking questions based on a set of seven categories that constitute each level of use: knowledge, acquiring information, sharing, assessing, planning, status reporting and performing. A matrix is constructed depicting the behaviour of the interviewee, indicating the level of use. A final assessment of an individual's overall level of use is made by considering responses at each of the decision points and by examining the behaviour matrix holistically. It is important to acknowledge that, in addition to being able to place individuals at different levels of use, more in-depth data were gathered through the focused interviews. In this study, to discover if participants have changed their behaviour as a result of the professional development program and to demonstrate the type of information the levels-of-use profile descriptors in the Concerns Based Adoption Model provide, only the levels-of-use generic descriptors provided by the model will be reported.
The level of use is considered a single item survey (Loucks et al., 1975) and therefore internal consistency measures are not appropriate. But test--retest reliability estimates have been found to fall in the range of 0.84 to 0.87 (Hancock, Knezek & Christensen, 2007). Additionally, a study by Hall & Loucks (1977) involving 1381 levels of use interviews revealed inter-rater reliabilities ranging from 0.87 to 0.96 demonstrating good to excellent consistency among raters. Some evidence for the validity of the levels of use was provided by Hall & Loucks (1977) in a study involving 45 teachers. Comparisons were made between ethnographic data gathered by researchers observing teachers in the field and the ratings assigned by the levels of use protocol. Correlation coefficients ranged from 0.65 to 0.98.
Data collection and analysis
For the current study, data were collected in two stages: administration of the Stages of Concern Questionnaire followed by the levels of use focused interview. To Help ensure consistency in the focus of responses across the two instruments, Participants were asked to choose a single innovation that they had acquired from the Instructional Intelligence professional development initiative, and to base Their responses on their implementation experiences for that innovation only. The questionnaire was administered using a secure online survey platform that saved responses to a spreadsheet. The levels of use interviews were conducted by the author on a one-on-one basis with each research participant. All interviews were audio recorded and field notes were also taken.
For each participant, individual profiles were generated from the questionnaire data that displayed relative intensities of concern for each stage. Data generated by the questionnaire can be analysed in two ways. The more common method of analysis is the 'peak stage concern'. This involves identifying the peak concerns of individuals and interpreting them in relation to the model's user profile descriptors. The second involves analysing the 'first- and second-highest stages' of concern in combination. Examining both highest and second-highest concerns provides additional insight into the dynamics of concerns and reveals general developmental patterns for both groups and individuals. For this study, both peak and combination of highest and second-highest stages of concern analysis were conducted. The data were also analysed to identify subgroups who had similar concerns. An overall group profile was generated by averaging all of the raw scores and converting them into percentiles, producing a 'typical group profile'. Interview data were transcribed and analysed against the levels of use categories. An assessment on an individual's overall level of use was made by considering responses to each of the decision points and by classifying behaviour holistically using the levels of use matrix. Finally, data were analysed to identify relationships between levels of use and stages of Concern profiles, and also to identify relationships to the overall group profile.
Stages of concern profiles: Peak-stage concern group results
Analysis of the questionnaire findings for the study group revealed that the Peak stage of concern was Stage 5--collaboration. These scores are shown in Table 4. Stage 5 concerns are 'impact' concerns (Hall & Hord, 2006) and relate to the Impact the use of the innovation is having on those around the user and on the wider system.
Group characteristics were defined using the Concerns Based Adoption Model user-profile descriptions as elaborated below.
Stage 5--collaboration concerns Twenty individuals or 74% of the group had peak Stage 5 concerns indicating that the majority of the group is concerned about collaborating with others in their use of the innovation. Hall & Hord (2006) suggested that collaboration concerns are 'very rare in any organisation, including schools' (p. 150) and when this occurs, 'something very special' (p. 137) has been done to manage and support the process. Concerns Based Adoption Model literature (Hall & Hord, 2006) states that if an innovation is appropriate and the change process, including professional development, is facilitated wisely, then implementers will move from self concerns (information and personal) to task (management) concerns within three years. At three to five years, they will progress to impact concerns (consequence, collaboration and refocusing). This developmental process is in line with the work of Fullan (2001), who suggested that once a decision has been taken to adopt or proceed with change (Phase I--initiation), the next stage (Phase II--implementation) or initial use usually takes two to three years (Concerns Based Adoption Model self and task concerns) and involves early attempts at putting the initiative into action. The transition to the final Phase III (institutionalisation--the model's impact concerns) refers to the change being Built into the system and becoming part of routine working practice. Fullan (2001) suggested that the time taken to move from initiation to institutionalisation takes between three and five years with reasonably complex initiatives and five to 10 years for large-scale efforts. For the majority of this group to have developed to collaboration 'means that change has truly been treated as a process, that the innovation has been given sufficient time to be implemented' (Hall & Hord, 2006, p. 150).
With regard to the amount of time individuals had participated in the professional development program, 12 individuals had taken part in all four years, 3 had been involved for three years and five had participated for two years. For three-quarters of the group (15) to be displaying impact concerns after three to four years of participation indicates that the program has been successfully implemented and participants were using different instructional innovations in their practice. For the remaining 5 participants to be at this level after only two years of participation in the program is significant and further research is need to establish what factors facilitated their rapid transition to this level.
Stages of concern profiles: Highest and second-highest concerns group results
As depicted in Table 5, a matrix was developed which cross-tabulated each individual's highest and second highest stage of concern.
Results for the questionnaire revealed that 74% of the group were concerned with collaborating with others about a range of issues. Further analysis (also examining the second-highest stage) indicates that the reasons individuals want to collaborate are varied and range across the full spectrum of concerns, from collaborating about any issue regarding use (Stage 0); gaining more information about the use of innovations (Stage 1); managing time and resources (Stage 2); considering the impact use has on students (Stage 4); to changing the way the innovation is used (Stage 6).
Analysis of subgroups
Analysis of subgroups using the questionnaire profiles reveals some interesting findings. A selection of illustrative cases is discussed below.
Highest stage of concern Stage 2 (personal) and second-highest stage of concern Stage 1 (information): two individuals These individuals have intense personal concerns, are uncertain about the innovation and are analysing their relationship towards use. When combined with Stage 1, this profile indicates that they are seeking more information about use. George and colleagues (2006) stated that this profile is indicative of early users who require more information and need to resolve personal concerns related to their status within the organisation. This profile is also known as a negative one-two split (George et. al., 2006). When Stage 2 concerns override Stage 1, individuals may have various degrees of self-doubt and resistance to use of the innovation use, suggesting that the person is preoccupied with the innovation's effect on their personal position or job security. These concerns are often so powerful that they overwhelm any desire to learn more and the individual is unable to deal with the innovation objectively until these concerns have been reduced. Both individuals had participated in the program for the full four years with one ranked at IVA Routine use and the other, IVB Refined use, so they are clearly implementing innovations. For them to have this stages Of concern profile after this length of time is uncommon, and the reasons are not clear. Concerns Based Adoption Model authors suggest an appropriate intervention strategy would be to discuss the issues in a generalised and non-threatening manner; this support often helps alleviate intense personal concerns and produces a more positive disposition to implementation.
Highest stage of concern Stage 2 (personal) and second highest stage of concern Stage 5 (collaboration): two individuals These individuals have intense personal concerns and uncertainties about the innovation and are analyzing their relationship towards the status and rewards of using the innovation within their organisation. As mentioned above, this profile is indicative of early users, and, combined with Stage 5, indicates that users would like to collaborate with others regarding these issues. Stage 2 concerns indicate uncertainty regarding implementation and additional research would be needed to discover the reasons for this. Neither participant had taken part in the full four years of the professional Development program; one had participated for two years, the other for three years. Examination of the levels of use interview data revealed that both participants were both Ranked at IVA Routine use but one had recently moved to a new position and was currently re-negotiating her role, something that could explain the profile. There was no indication of any change to the second individual's role or status, Indicating a need to gather additional data to explain the result.
Highest stage of concern Stage 3 (management) and second highest stage of concern Stage 5 (collaboration): two individuals High Stage 3 (management) concerns indicate that these individuals are experiencing difficulty implementing the innovation and are preoccupied with task-related issues, such as organising their time and the resources related to implementation. When combined with the second-highest concern Stage 5 (collaboration), this suggests they would be interested in working with others to discuss their concerns and possibly obtain assistance. Both individuals had participated in the full four years of the program and additional information would be required to establish what specific task concerns these individuals are experiencing.
Highest stage of concern Stage 6 (refocusing) and second-highest stage of concern Stage 2 (personal): one individual This individual had participated in all four years of the program and is focused on exploring new ways of using the innovation or has ideas about improvements, including the possibility of making major changes to it or replacing it. Hord and colleagues (2004) issued a word of caution regarding this profile, as it could indicate the person may be considering a return to old practices and, unless something changes, may abandon use of the innovation. George and colleagues (2006, p. 54) stated that when a Stage 6 profile 'tails up' it can represent a negative attitude towards the innovation or may suggest non-use. Combined with high Stage 2 concerns, the profile becomes even more intriguing. Personal concerns are typical of non-users and indicate intense concerns, often pointing to an uneasiness regarding use. Analysis of the levels of use data revealed this individual was indeed a user but the profile certainly warrants further investigation to establish the issues that contributed to it.
Levels of use
Findings revealed that all the teachers involved in this study were implementing the innovations in their practice as a result of the Instructional Intelligence professional development program. Three distinct groups were identified in the sample (Table 4).
Group characteristics were defined using the Concerns Based Adoption Model user profile descriptions and are elaborated in Table 6.
Level of use IVA--routine: 11 individuals (41%) Hall & Hord (2006, p. 13) stated that a 'lack of change' in the way an individual is using an innovation is the key to identifying a routine user. Having mastered use, routine users establish a regular pattern of working with the innovation and have no plans to adapt or change. While placement at this level provides information about an individual's type of use, it is not clear if he or she has changed their use over the fouryear program period or if they have made a recent change and are waiting to see the effects. The relationship between the number of years individuals have participated in the program and their placement in this category provides additional information about an individual's progress through the levels of use and the implementation of the professional development program. Five individuals in this group had participated in the program for 4 years, two for 3 years and four for 2 years. These findings suggest that it is possible for individuals to become routine users of An innovation within two years; Hall and Hord (2006, p. 172) suggested that to move to this higher level of use such individuals need to 'have had appropriate facilitative assistance and time' and it is indicative of the change initiative being implemented appropriately. But additional investigations are required to establish the different characteristics of use and the length of time established patterns have been operating.
Level of use IVB--refinement: 14 individuals (52%) This group is actively involved in planning or making amendments to the way they use the innovation to improve student outcomes. To be placed at this level, individuals must have changed within the past three months, be planning a change or be in the process of changing or evaluating use. Hord and colleagues (2004) stated that these individuals make excellent models for other users (including those who may be struggling) and should be encouraged to share their ideas and demonstrate their use of the innovation. A key way to support this group is to provide opportunities for collaboration with others using the same innovation, to foster new ideas and reinforce use. This information is particularly valuable when read in conjunction with the questionnaire data, which revealed the majority of the group were placed at Stage 5 (collaboration). Most of the group do indeed wish to collaborate; providing opportunities for them to work together would be an appropriate support strategy at this point in time. The majority of this group (10 individuals) had participated in the program for four years, two for three years and two for two years. These findings demonstrate that is it is possible for individuals to reach higher levels of use after two years of program participation, although the proportion in this category is smaller than the number in 'routine' use. Hall and Hord (2006) stated that there is a correspondence between levels of use and stages of concern, in that a non-user is likely to have more intense self concerns and it would be reasonable to assume that an individual with higher levels of use will have impact concerns. But they caution against making assumptions and stress that, despite the attractiveness of such a relationship, there is no research to support a correlation and it oversimplifies the complex dynamics of change. Additional information is required here to establish individual issues.
Level of use V--integration: two individuals (7%) Both the individuals in this group had participated in the program for the full four years. Placement in this group indicates that they have moved beyond personal use to work with others to coordinate their efforts for the purpose of improving student outcomes. Any changes being made do not relate to merely disseminating information about an innovation but instead focus on increasing the impact on students and must benefit the students of all collaborators. Movement to the higher levels of use is not always possible or suitable for all individuals and Hord and colleagues (2004) affirmed that individuals within this group are in a unique position to be able to influence the change effort and support others on their change journeys. In this sense this group require 'special consideration' (Hord et al., p. 68). Once more, the provision of structured collaborative activities for this group and their colleagues would be appropriate.
The aim of this article is to describe and examine the use of the Concerns Based Adoption Model as a framework for the assessment of professional development initiatives in the VET sector using examples from the recent implementation of the Instructional Intelligence program in Western Australia. These examples illustrate the use of the model and its potential for strengthening the evaluation and assessment of professional development initiatives in the Australian VET sector. Specifically, these examples illustrate the use of the model as a means of answering questions such as how individuals respond to professional change initiatives and what support mechanisms best suit their needs at different stages of the change process. This information can be used to adjust the support provided in the current program, as well as to inform the design of future models of professional development that meet the needs of individual VET teachers operating in dynamic systems.
In this study, the use of the model (the Stages of Concern Questionnaire and the Levels of Use Focused Interview) provides insight into how change has been experienced as a result of the Instructional Intelligence professional development program and to what degree changed practice has been implemented by individuals. The example findings indicate that teacher participants in the professional development initiative are implementing instructional innovations in their practice at routine, refined and integrated levels of use: learning has been transferred into practice. Results for the stages of concern also reveal that 74% of the group are concerned with collaborating on issues ranging from gaining more information about use and managing time and resources to further investigating the impact that innovations have on their students' learning. These types of findings are particularly helpful in assessing the progress of change as a result of professional development. In addition, and arguably more importantly in a systemic approach to professional development, knowledge of concerns and levels of use can usefully inform the design and provision of ongoing support or intervention strategies targeted to meet the specific needs of both individuals and the overall group (Hall & Hord, 2006; Hall & Loucks, 1979). For example, the findings indicate that the majority of the participants were interested in sharing their experiences; this knowledge was used by developers to establish a 'community of practice', which thereby met the group's most immediate need: opportunities for further collaboration.
Knowledge of individual and group levels of use is also valuable for those responsible for managing the change process; this information can be used to help establish peer coaching relationships and help reduce the feelings of isolation often experienced by implementers. For example, individuals focusing on the use of specific innovations could be matched with others using the same instructional strategies in their classrooms. Working together they could exchange ideas about the different ways in which they are using the innovation with their students, share resources and provide collegial support to each other. Additionally, an individual who has had several years' experience implementing a particular innovation or aspect of change could act as a mentor to another who may be struggling or who is just about to embark upon the process.
The conceptual framework that underpins the Concerns Based Adoption Model also proved helpful in better understanding how change is mediated at the individual and system levels. Research that underpins the model indicates that, when change is implemented and supported appropriately, individuals move from self- to task-related concerns within three years, and from task- to impact-related concerns within three to five years (Hall & Hord, 2006). Combining the Concerns Based Adoption Model with another theoretical framework of change (for example, Fullan, 2001) provided an additional lens through which to investigate and to corroborate assumptions about the dynamics of change further. For example, Fullan's theoretical model suggests that it takes three to five years for change To move from 'initiation' to 'institutionalisation' for reasonably complex change initiatives. Examining change from different theoretical perspectives offers comparable views; these lenses provide additional insight into the process and in turn contribute to our understanding of how different theoretical perspectives can be aligned to deepen our knowledge. In turn, this can add to the further development of theories of change (Anderson, 1997; Fullan, 2006; Hargreaves, 2005; Hargreaves & Fullan; 2009).
While the findings generated through the use of the Concerns Based Adoption Model in this study are generally positive and potentially useful to Professional development managers and developers, there are also some limitations of the data that readers should note. First, the model's instruments were applied in this study to provide a cross-sectional 'snapshot' of the study group at a particular point in time. But this model's instruments could be used at several points in a longitudinal study to obtain a picture of the developmental pattern of change experiences, thereby helping to strengthen understanding of the process of change (Anderson, 1997; Hargreaves, 2005). This would still not reveal why individuals are at different stages of concern and levels of use, or how they got there. For example, it is unclear what elements mediated use, or what supported or hindered individual progress. In order to better understand the wider systemic issues that affect individuals and their unique experiences, additional data needs to be collected. But the model does provide a research-based framework to understand the change process, and its instruments generate solid empirical data that can serve as a starting point from which to further explore the systemic issues that contribute to individual placements. Second, the professional development participants who volunteered to take part in this research study were motivated and interested to do so and it is not surprising that they are implementing aspects of the program. It is also the case that variations in levels of use and stages of concern exist amongst the group and that findings based on the Concerns Based Adoption Model can still be used to help us better understand the group's experience of change associated with professional development. The volunteer nature of the research participants does not diminish the value of using this model for helping us better understand wider change initiatives.
As the AustralianVET sector, like many others in the international community, embarks on meeting the challenges of the 21st century, planned, proactive approaches to building and sustaining change and to those who make it are urgently required (Guthrie & Clayton, 2010; OECD, 2008, 2009; Skills Australia, 2011). Without effective planning and appropriate resources, any change effort is likely to fail (Hall & Hord, 2006). While a solid plan is required at the start of the process to help to articulate a shared vision and to outline the steps that need to be taken, the planning process should not be a one-off event, and neither should plans be rigid and inflexible. Resource requirements change as individuals become more expert in their implementation of change, as circumstances change, or if they are suddenly required to respond to and integrate the latest policy reform or curriculum mandate. In order to support the change process effectively, reliable data are required to help capture how individuals negotiate and respond to change-related issues.
Ultimately, change is a multidimensional process and characteristically a long and messy business. For teachers to become effective at facilitating sustainable change, these features need to be kept in mind; the use of valid and reliable methods can help to develop a better understanding of these processes. Incorporating mechanisms to systematically monitor and assess the complexities of the process, particularly in the dynamic environment of VET, is vital when implementing suitable intervention strategies (Cort et al., 2004; OECD, 2008, 2009). The design and implementation of this process needs to be informed by rigorous, evidence-based assessment and evaluation practices. While it is acknowledged that the Concerns Based Adoption Model has limitations, the findings in this study have revealed that it can be useful in assessing the impact of a professional development initiative for VET teachers. These findings can be used to not only help inform the future design and implementation of programs, but also (and arguably most importantly) to support those who are at the forefront of making the changes demanded by the sector--VET teachers.
Anderson, S. E. (1997). Understanding teacher change: Revisiting the concerns based adoption model. Curriculum Inquiry, 27(3), 331-367.
Attwell, G. (1997). New roles for vocational education and training teachers and trainers in Europe: A new framework for their education. Journal of European Industrial Training, 21(6/7), 1-12.
Australian National Training Authority. (1997). Research reports into professional development. Brisbane: author.
Bennett, B. (2002). Instructionally Intelligent ... socially smart. Orbit, 32(4), 1-5.
Bennett, B. (2010). The artful science of instructional integration. In R. Marzano (Ed). On excellence in teaching (pp. 65--91). Bloomington, IN: Solution Tree Press.
Bennett, B., & Rolheiser, C. (2006). Beyond Monet: The artful science of instructional integration. Toronto, Ontario: Bookation Inc.
Caven, P. (2004). Report on a strategic evaluation of Reframing the Future. Brisbane: Australian National Training Authority.
Corcoran, T. B. (1995). Transforming professional development for teachers: A guide for state policymakers. Washington, DC: National Governors' Association.
Corcoran, T. B., Shields, P. M., & Zuker, A. A. (1998). Evaluation of NSF's statewide systemic initiatives (SSI) Program: The SSIs and professional development for teachers. Menlo Park, CA: SRI International.
Cort, P., Harkonen, A., & Volmari, K. (2004). PROFF--Professionalisation of VET teachers for the future. European Centre for the Development of Vocational Training (CEDEFOP). Panorama Series 104. Luxembourg: Office for the Official Publications for the European Communities.
Cummings, T. G., & Worley, C. G. (2008). (Organization development and change. Manson, OH: Cengage Learning.
Dickie, M., Eccles, C., Fitzgerald, I., McDonald, R., Cully, M., Blythe, A., Stanwick, J. & Brooks, L. (2004). Enhancing the capability of VET professionals project: final report. Brisbane: Australian National Training Authority.
Ellsworth, J. B. (2000). Surviving change: A survey of educational change models. New York, NY: ERIC Clearinghouse on Information and Technology, Syracuse University.
European Centre for the Development of Vocational Training (CEDEFOP). (2009). Modernising vocational education and training. Fourth report on vocational education and training research in Europe: Synthesis report. Luxembourg: Publications Office of the European Union.
European Centre for the Development ofVocational Training (CEDEFOP), (2010). A bridge to the future: European policy for vocational education and training 2002-10. Luxembourg: Publications Office of the European Union.
Figgis, J. (2009). Regenerating the Australian landscape of professional VET practice: Practitioner driven changes to teaching and learning. Adelaide: National Centre for Vocational Education Research (NCVER).
Figgis, J., & Hillier, Y. (2009). Taking 'innovation' on the road. An Occasional Paper. Adelaide: National Centre for Vocational Education Research (NCVER).
Forewood, A., Mclean, B., & Butler, J. (2001). Defining and scoping human resource development in the Victorian TAFE system. Melbourne: Office of Employment, Training and Tertiary Education.
Fullan, M. (2001). The new meaning of educational change. New York, NY: Teachers College Press, Columbia University.
Fullan, M., & Hargreaves, A. (Eds). (1992). Teacher development and educational change. Oxford, UK: RoutledgeFalmer.
Fuller, F. F. (1969). Concerns of teachers: A development conceptualization. American Educational Research Journal, 6(2), 207-226.
Garet, M. S., Porter, A. C., Desimone, L., Birman, B. F., & Suk Yoon, K. (2001). What makes professional development effective? Results from a national sample of teachers. American Educational Research Journal, 38(4), 915-945.
George, A. A., Hall, G. E., & Stiegelbauer, S. M. (2006). Measuring implementation in schools: The stages of concern questionnaire. Austin, TX: SEDL.
Guskey, T. R. (2002). Professional development and teacher change. Teachers and Teaching: Theory and Practice, 8(3), 381-391.
Guthrie, H. (2010). Professional development in the vocational education and training workforce. Adelaide: National Centre for Vocational Education Research (NCVER).
Guthrie, H., & Clayton, B. (2010). Building capability in vocational education and training providers:
The TAFE cut. Adelaide: National Centre for Vocational Education Research (NCVER). Guthrie, H., Perkins, K., & Nguyen, N. (2006). The future now 2006-2010: The roles, knowledge and skill requirements of the VET practitioner. Perth: Western Australian Department of Education and Training.
Hall, G., George, A., & Rutherford, W (1998). Measuring stages of concern about the innovation: A manual for use of the SoC Questionnaire. Austin, TX: Southwest Educational Development Laboratory.
Hall, G., & Hord, S. (1987). Change in schools: Facilitating the process. Albany, NY: State University of New York Press.
Hall, G., & Loucks, S. (1979). Implementing innovations in schools: A concerns-based approach. Research and Development Centre for Teacher Education. Austin, TX: University of Texas.
Hall, G. E., Dirksen, D. J., & George, A. A. (2006). Measuring implementation in schools: Levels of use. Austin, TX: SEDL.
Hall, G. E., George, A. A., & Rutherford, W. L. (1979). Measuring the stages of concern about the innovation: A manual for the use of the SoC Questionnaire (Report 3032). Austin, TX: Research and Development Center for Teacher Education, University of Texas.
Hall, G. E., & Hord, S. M. (2006). Implementing change: Patterns, principles and potholes. New York, NY: Pearson Education.
Hall, G. E., & Loucks, S. F. (1977). A developmental model for determining whether the treatment is actually implemented. American Educational Research Journal, 14(3), 263-276.
Hancock, R., Knezek, G., & Christensen, R. (2007) Cross-validating measures of technology integration: A first step toward examining potential relationships between technology integration and student achievement. Journal of Computing in Teacher Education, 24(1), 15-21.
Hargreaves, A. (Ed.). (2005). Extending educational change: International handbook of educational change. Netherlands: Springer.
Hargreaves, A., & Fullan, M. (Eds). (2009). Change wars. Bloomington, IN: Solution Tree. Harris, R., Simons, M., & Clayton, B. (2005). Shifting mindsets: The changing work roles of vocational education and training practitioners. Adelaide: National Centre for Vocational Education Research (NCVER).
Harris, R., Simons, M., Hill, D., Smith, E., Pearce, R., Blakeley, J., Choy, S., & Snewin, D. (2001). The changing role of staff development for teachers and trainers in vocational education and training. Adelaide: National Centre for Vocational Education and Research (NCVER).
Hord, S., Rutherford, W L., Huling, L., & Hall, G. E. (2004). Taking charge of change. Austin, TX: SEDL.
Huberman, M. (1983). Recipes for busy kitchens. Knowledge: Creation, Diffusion, Utilization, 4(4), 478-510.
Huberman, M., & Miles, M. (1984). Innovations up close. New York, NY: Plenum.
Joyce, B., & Showers, B. (1995). Student achievement through staff development. New York, NY: Longman.
Joyce, B., & Weil, M. (1996). Models of teaching (5th ed.). New York, NY: Allyn & Bacon.
Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating training programs (3rd ed.). San Francisco, CA: Berrett-Koehler Publishers.
Kotter, J. (1996). Leading change. Boston, MA: Harvard Business School Press.
Lieberman, A., & Pointer Mace, D. H. (2008). Teacher learning: The key to educational reform. Journal of Teacher Education, 59(3), 226-234.
Loucks, S. F., Newlove, B., & Hall, G. E. (1975). Levels of use of the innovation: A manual for trainers, interviewers, and raters. Austin, TX: Research and Development Center for Teacher Education, University of Texas.
Loveder, P. (2005, August). World trends in staff development: Implications on the performance of technical education institutions. Paper presented at Pan Pacific KLLA National Seminar: The Development of Technology and Technical Vocational Education and Training in a Era of Globalisation, National Centre for Vocational Education Research (NCVER), Adelaide.
McDonald. R. (1999) Towards evidence-based vocational education and training. Adelaide: National Centre for Vocational Education Research (NCVER).
Mahlamaki-Kultanen, S., Susimetsa, M., & Ilsley, P. (2006). Leonardo da Vinci projects supporting the changing role of VET teachers and trainers. Helsinki, Finland: Centre for International Mobility, Finnish Leonardo National Agency
Mitchell, J., Chappell, C., Bateman, A., & Roy, S. (2006). Quality is the key: Critical issues in teaching, learning and assessment in vocational education and training. Adelaide: National Centre for Vocational Education Research (NCVER).
Mitchell, J., & Ward, J. (2010). The JMA analytics model of VET capability development: A report on the national survey of vocational education and training (VET) practitioner skills, conducted October--November 2009. Sydney: John Mitchell and Associates. Retrieved from http://www.jma.com.au/User_Uploaded_Files/ file/JMA%20Analytics%20National%20VET%20Practitioner%20Skills%20Report%2031%2 0January%202010.pdf
Moynagh, M., & Worsley, R. (2003). Learning from the future: Scenarios for post16 learning. Hertford, UK: Learning Skills Research Centre.
Organisation for Economic Co-operation and Development. (2008). Learning for jobs: OECD reviews of vocational education and training--Australia. Paris: author.
Organisation for Economic Co-operation and Development. (2009). OECD/CERI study of systemic innovation in VET: Systemic innovation in the Australian VET system: Country Case Study Report. Paris: author.
Productivity Commission. (2011). Vocational education and training workforce. Research report. Retrieved from http://www.pc.gov. au/_data/assets/pdf_file/0009/108369/vocational-workforce.pdf
Perkins, K. (1997). Lessons learnt: An analysis of findings of recent evaluation reports on professional development in VET. Research reports into professional development. Brisbane: ANTA.
Perkins, K., Guthrie, H., & Nguyen, N. (2006). Towards a learning culture: The impact of the TLASG program on the VET sector in Western Australia. Unpublished report, Department of Education and Training, Perth.
Rauner, F., & Maclean, R. (Eds.). (2008). Handbook of technical and vocational education and training research. New York, NY: Springer.
Senge, J. E. (1990). The fifth discipline: The art and practice of the learning organization. New York, NY: Doubleday.
Schofield, K., & McDonald, R. (2004). Moving on: Report of the high level review of training packages. Melbourne: Australian National Training Authority.
Showers, B., Joyce, B., & Bennett, C. (1987). Synthesis of research on staff development: A framework for future study and a state-of-the-art analysis. Educational Leadership, 45(3), 77-87.
Skills Australia. (2010). Australian workforce futures: A national workforce development strategy. Canberra: Commonwealth of Australia.
Skills Australia. (2011). Skills for prosperity--a roadmap for vocational education and training. Canberra: Commonwealth of Australia.
Smith, A., Courvisanos, J., Tuck, J., & McEachern, S. (2012). Building the capacity to innovate: The role of human capital. Adelaide: National Centre for Vocational Education and Research (NCVER).
Unger, Z., & Rutter, A. (1997, October). Strategic training evaluation. Paper presented at Australasian Evaluation Society, Curtin, proceedings, Evaluation: Equipping communities and government, Adelaide.
Villegas-Reimers, E. (2003). Teacher professional development: An international review of literature. Paris: UNESCO, International Institute for Educational Planning.
Volmari, K., Helakorpi, S., & Frimodt, R. (Eds.). (2009). Competence framework for VET professions: Handbook for practitioners. Sastamala, Finland: Finnish National Board of Education.
Wheelahan, L., & Moodie, G. (2010). The quality of teaching in VET: Final report and recommendations. Canberra: Department of Education, Employment and Workplace Relations.
Wilson, D. (Deborah Wilson Consulting Services). (2003). Final report: Strategic professional development project: strategic analysis of the professional development needs of vocational education and training professionals/stakeholders in Queensland. Brisbane: Department of Employment and Training, Queensland.
Rebecca Saunders is a PhD candidate in instructional intelligence and educational change at the School of Education, Murdoch University. Email: firstname.lastname@example.org
Table 1 Participants by years of teaching experience and years of program participation Number of years of Instructional Intelligence Teaching experience program (years) participation Male Female Metro 1-4