Verifying the Efficacy of Vocational Guidance Programs: Procedures, Problems, and Potential Directions
Perry, Justin C., Dauwalder, Jean Pierre, Bonnett, Heather R., Career Development Quarterly
This article summarizes 12 presentations in Group 7 of the 2007 joint symposium of the International Association for Educational and Vocational Guidance, Society for Vocational Psychology, and National Career Development Association held in Padua, Italy, that focused on procedures for verifying the efficacy of vocational guidance programs. Three themes, or general tensions, emerged from the group's discussion of the presentations: (a) quantitative versus qualitative evidence, (b) public policy needs versus local needs, and (c) comprehensive versus cost-effective programs. On the basis of these core themes, recommendations are made for 3 potential directions for research that address certain threats and opportunities for the field.
Conscious of an era emphasizing evidence-based practice and empirical standards of accountability, Group 7 of the 2007 joint symposium of the International Association for Educational and Vocational Guidance, Society for Vocational Psychology, and National Career Development Association held in Padua, Italy, focused on procedures that can enable practitioners, consumers, and stakeholders to evaluate the effectiveness of vocational guidance programs. In other words, how do practitioners know that vocational interventions achieve their intended outcomes? What are the criteria that should be adopted to judge the credibility of methods used for measuring such outcomes? After 12 presentations by scholars and practitioners from the international community of vocational guidance practitioners, the group's discussion led to more questions than answers. In this article, we summarize three thematic tensions that emerged during the course of the discussion. We also discuss threats, which were identified by the group participants, to the sustainability and adaptation, within shifting cultural contexts, of vocational guidance programs that are responsive to the needs of diverse populations across diverse settings.
Theme 1: Quantitative Versus Qualitative Evidence
Perhaps it was unavoidable that a tension between quantitative and qualitative methods of program evaluation would emerge. This tension became evident when Steven Brown of the United States responded, during the presentation of a paper (S. D. Brown, 2007), to a question about how researchers should demonstrate the efficacy of vocational guidance programs. His recommendation was to conduct a meta-analysis. S. D. Brown clarified that his idea should not be taken as seriously as it might have come across; yet, the authence thought that it was a crucial point, for it brought into play the relative priority that scholars should place on deductive, statistical findings versus contextualized, inductive findings. Although meta-analyses can help to establish the overall effectiveness of interventions, their relevance may be limited in guiding programs to meet the specific concerns of populations presenting with a constellation of risk factors and/or cultural matters that do not correspond to (or are not represented by) the prevailing samples, problems, and constructs on which the meta-analyses are based. Indeed, this limitation has previously been noted by Whiston (2002) concerning efficacy literature as a whole among career counseling interventions.
Cinamon (2007) highlighted the challenges of merging standardized quantitative outcome measures with locally constructed narratives in her study of Israeli youth at risk for school dropout (i.e., Russians, Ethiopians, Israeli Arabs) across seven high schools. In her school-towork project, which was designed to help schools build the capacity to carry out their own guidance programs, the author noted that common outcome variables, such as career decision-making self-efficacy, do not necessarily offer a useful assessment of program efficacy because they do not represent the same meanings to different children, depending on their ethnic experiences. Cinamon suggested that important outcomes of interest should not be restricted to quantitative measures, but should be complemented by qualitative findings, which, in her research, illustrated the value of assessing meanings and motivations that derive from outcome expectations. …