Academic journal article Journal of School Health

Illuminating Negative Results in Evlauation of Smoking Prevention Programs

Academic journal article Journal of School Health

Illuminating Negative Results in Evlauation of Smoking Prevention Programs

Article excerpt

Among the more cogent criticisms of conventional quasiexperimental program evaluations, in health as well as in other domains, is that negative and especially null results usually give program planners little empirical insight about what went wrong and how best to fix it. (1,2) Research tactics complementary to conventional program evaluation design can help illuminate negative results.

These complementary tactics focus on the evaluation of program implementation processes. In a conventional evaluation of a smoking prevention program, children assigned to receive the intervention may be compared with children assigned to a control group. Because interventions are not always implemented exactly as planned, it may be misleading to leave unexamined the assumption that the independent variable in this simple evaluation design, that is the presence of absence of a particular set of activities delivered in a particular way, represents what actually occurred. If no differences between groups are detected, there are actually three rather than the usual two possible explanations: no program effect, Type II error (failure to detect a true program effect), or failure of program implementation. The latter possibility, termed by some authors Type III error, (3,4) essentially amounts to a specification error in the independent variable. It deserves attention because insofar as sources of mis-specification are identifiable, hypotheses about what went wrong in implementation can be generated and used to guide and inform future interventions. Thus, systematic assessment of the sources of natural variation in implementation across intervention sites can represent important complements to conventional impact evaluation. (5-8)

Previous investigations have focused on quantifying the extent of program implementation. (3,4,8) This approach may fail to adequately illuminate reasons for implementation problems. Research on implementation (2,6,9) has shown that peculiarities of local contexts are the most important determinants of deviation from planned implementation. This factor suggests evaluation of program implementation should examine how programs come to be "enacted" (10) within the local configuration of human and material resources, organizations, and characteristics. (2,11) Such investigation is perhaps most effectively conducted with naturalistic research methods rather than highly quantitative normative techniques. (12-14) Though sometimes more costly to collect and to analyze, naturalistic data can be subjected to the same standards of rigor as quantitative data, (15) while providing richer detail about local situations and enhancing their potential utility.

In this project, conventional quasiexperimental evaluations of three school-based smoking prevention programs for children produced no statistically significant effects on children's knowledge, attitudes, intentions, or behavior. However, complementary evaluations of the program implementation process conducted using naturalistic methods suggested reasons for null effects were different at each site. This description of the programs and their evaluations focuses on how complementary implementation evaluation tactics facilitated this conclusion.


Three smoking prevention programs were pilot-tested for inclusion in a communitywide intervention project aimed at decreasing cardiovascular disease risk in a socioeconomically disadvantaged Montreal neighborhood. (16,17) Because the objectives of the pilot-testing were to assess how effectively programs could prevent smoking and how easily they could be implemented, evaluation plans called for both impact and implementation evaluation at all three sites. Complying with recommendations of a community advisory committee, programs were selected for implementation in day cares, primary schools, and high schools.

The day care program, "Vivre sans fumee," was designed to sensitize children ages three-six to the unpleasant aspects of tobacco smoke while reinforcing a positive image of smoke-free environments. …

Search by... Author
Show... All Results Primary Sources Peer-reviewed


An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.