This article argues that researchers entrusted with evaluating the educational outcomes of African American children must engage their practice for the public good and on behalf of African American students. This can be done by collecting and reporting data that will assist in the creation of more equitable school reform for elevating the achievement of African American students. In this article, historical perspectives and current trends in evaluation research are illuminated. Using the Howard University Evaluation Training Institute as a guide, the steps for conducting quality evaluations are described, and the importance of conducting culturally and contextually responsive evaluations are highlighted.
Educators are currently in an era of intense accountability and efforts to elevate the achievement of all students, but particularly those placed at risk for underachievement and academic failure. In response to demands from the general public, employers, and policymakers that the nation achieve a higher level of learning in the K-12 schools, there have been serious attempts to reform public education. These demands have been fueled by the low achievement and high dropout rates of many high school students, particularly those who are low income and from disadvantaged groups. For the class of 2003-2004, for example, only 73.9% of all public school students graduated on time with a regular high school diploma four years after starting 9th grade (Laird, Lew, DeBell, & Chapman, 2006). Furthermore, there is a disparity in the graduation rates of White and ethnic minority students, with the exception of Asians. Using data from the U.S. Department of Education, Greene and Winters (2005) found that for the class of 2002, about 78% of White students graduated from high school with a regular diploma, compared to only 56% of African American students and 52% of Hispanic students. Additionally, these data revealed that only 40% of White students, 23% of African American students, and 20% of Hispanic students, who started public high school, graduated college-ready-that is, with the minimum set of skills and credentials required to attend a four year college.
Recent passage of the No Child Left Behind Act of 2001 (NCLB, 2002) has significantly escalated the need for evaluation research to assess and understand the quality and value of educational interventions for all students and for evaluators to diligently work in the public interest. By requiring scientifically based research to justify funding for educational programs and activities, NCLB, in part, sought to motivate researchers, education companies, and others to take responsibility for investing in evaluations and clinical trials of educational interventions (Olson & Viadero, 2002).
This article will (a) discuss the purpose of evaluation and provide historical perspectives and current trends in evaluation research, (b) highlight the importance of evaluating educational efforts in the public interest, in general, and on behalf of African American children, in particular, (c) describe the steps for conducting quality evaluations utilizing, as a guide, the conceptual framework of the Howard University Evaluation Training Institute (HU/ETI), a National Science Foundation (NSF) funded project, and (d) draw special attention to how researchers engaged in evaluation of educational efforts targeting children of color must ensure that their work is not only technically sound, but culturally and contextually responsive to the needs of the population being served.
EVALUATION RESEARCH: PURPOSES, HISTORICAL PERSPECTIVE, AND CURRENT TRENDS
Program evaluation involves the use of social science research methods to examine a program's goals, objectives, outcomes, and impact. Evaluation research is also used to investigate a program's structure, characteristics, activities, organization, and political and social environment. The practice of evaluation has been eloquently described as one that can enable society to meaningfully learn about its persistent social problems and how to effectively solve them (Cronbach, Ambron, Dornbusch, Hess, Hornik, Phillips, Walker, & Weiner, 1980).
Although the historical roots of evaluation research can be traced back to the 17th century, the conduct of systematic and rigorous evaluation research is a relatively modern 20th century development (Rossi, Lipsey, & Freeman, 2004). It was not until the 1960s that there was widespread use of the social science research method to evaluate social programs. Probably the strongest stimulus to the growth of evaluation research in this country was the social programming initiated by President John F. Kennedy and expanded under President Lyndon B. Johnson's Great Society legislation or the federal war on poverty and racial injustice programs. During the 1960s, numerous social programs were launched in education, housing, health, criminal justice, and income maintenance with the goal of buffering the negative effects of poverty and racial injustice for individuals, families, and communities. A huge impetus for educational evaluation, in particular, came with Senator Robert Kennedy's 1965 rider to the Title I (compensatory education) section of the Elementary and Secondary Education Act (ESEA, 1965), which mandated and provided federal support for the evaluation of programs for disadvantaged students (Shadish, Cook, & Leviton, 1991). The rapid growth of federal funding for social programs, media reports of fraud, abuse, and mismanagement, and apprehensions about the legitimacy of social welfare programming led many in the U.S. Congress to push for more oversight and accountability of these efforts (Shadish, Cook, & Leviton, 1991). Thus, federal mandates for evaluations spread to numerous types of social programming.
During the 1970s, evaluation emerged as an applied research specialty that examined programs and interventions in areas such as education and public health. Evaluation research was conducted, in large part, to yield "hard data" that would provide sound information for planning more effective programs and contribute to recommendations for action to improve some aspect of the human condition. Evaluation was thus seen as a process rather than a product and the evaluation was considered successful to the extent that the information it generated became part of the decision-making process (Cooley & Lohnes, 1976). Efforts to professionalize the field of evaluation research continued to expand during the 1970s. To help improve the quality of evaluations, in 1975, the Joint Committee on Standards for Educational Evaluation, a coalition of major professional associations, was formed to set standards for those who design, commission, conduct, and use evaluations. This committee published three sets of evaluation standards for use in a variety of educational and other settings, including: (a) The Student Evaluation Standards (Joint Committee on Standards for Educational Evaluation, 2003), (b) The Program Evaluation Standards, 2nd edition (Joint Committee on Standards for Educational Evaluation, 1994), and (c) The Personnel Standards (Joint Committee on Standards for Educational Evaluation, 1988). Each of the standards developed by the Joint Committee can be placed in one of four fundamental categories to promote educational and other evaluations that are proper, useful, feasible, and accurate.
Scholarship in evaluation has proliferated. In 1976, the first scholarly journal in evaluation, Evaluation Review, was published by Sage Publications. Currently, there are at least a dozen scholarly journals devoted primarily to evaluation, for example, New Directions for Evaluations, The American Journal of Evaluation, Practical Assessment, Research & Evaluation, Evaluation and Program Planning, and Educational Evaluation and Policy Analysis.
In 1986, the Evaluation Network and the Evaluation Research Society merged to create the American Evaluation Association (AEA). Currently, the AEA is an international professional association of approximately 4,000 evaluators devoted to the application and exploration of program evaluation, personnel evaluation, technology, and many other forms of evaluation in various areas including education.
Evaluation research, as a discipline, is rapidly evolving in the U.S. at federal, state, tribal, and local levels of government, as well as in non-governmental organizations and academia. Today, evaluation in education, in particular, is a very deliberate research endeavor which can yield information of use for myriad purposes. These include the use of evaluation data to (a) help shape the design and implementation of new and existing educational programs, (b) improve the management of existing educational programs and strategies, and (c) influence the development of educational policies and practices. Today, evaluation research is much more multidisciplinary and multifaceted in nature, drawing its theoretical foundations from various fields with a diverse menu of evaluation choices. It is also methodologically diverse with the evaluator relying on the full array of social science methods from the fields of psychology, statistics, education, sociology, political science, anthropology, and economics. Michael Quinn Patton, a noted evaluation scholar, identified more than 100 possible evaluation approaches that evaluators can select from depending on their particular situation and information needs (Harvard Family Research Project, 2002).
Two other current notable trends in evaluation research are worth mentioning. One is the shift from the quantitative versus qualitative debate to a position that recognizes that mixed methods or triangulation of perspectives in evaluation research has much to offer. Mixed-methods research can answer evaluation questions that neither quantitative nor qualitative research, in isolation, can answer. Furthermore, mixed-methods approaches provide opportunities for presenting a greater variety of divergent views (Teddlie & Tashakkori, 2003). Triangulation in evaluation can be done in multiple ways, including: (a) investigator triangulation, (b) multiple operationalism, (c) methodological triangulation, (d) target person triangulation, and (e) analysis triangulation (See Thomas, 2004, for greater discussion on triangulation in urban school reform efforts). The second notable trend in evaluation research is serious attention to issues of culture and context in evaluation planning, implementation, and reporting of findings. While this issue will be discussed later in this article, it should be pointed out that as the U.S. becomes increasingly more multicultural, evaluators must engage their practice in a manner that reflects the culture of program participants and stakeholders, thus, allowing stories and experiences to be told from these individuals' perspectives and culture (Thompson-Robinson, Hopson, & SenGupta, 2004).
EVALUATION IN THE PUBLIC INTEREST AND ON BEHALF OF AFRICAN AMERICAN STUDENTS
Evaluation research in the field of education, while often discussed as primarily a "methods-driven" activity, is not simply a scientific endeavor in search of "truth" and "solutions." Evaluation is often less concerned about general truths, because it focuses on specific programs and practices. Learning about program activities and outcomes often requires more than one way of thinking and full evaluator engagement in learning about the phenomenon under study (Rallis & Rossman, 2003). Evaluation is also very much a social enterprise that is best understood by taking into consideration the social, cultural, economic, and political contexts surrounding the program under consideration. Attention to public interest and public good are critical aspects of the evaluation process. Waters (1998) argues that evaluators of educational programs cannot ignore the fact that they become a part of the never-ending struggle to make judgment calls about a social activity which creates the conditions or obstacles for social mobility.
The Guiding Principles for Evaluators, developed and endorsed by the AEA in 1994 and revised in 2004, clearly articulates that researchers engaged in evaluation of programs and services have obligations that encompass work in the public interest, and that threats to the public good should never be ignored (AEA, 2004). This is the case, irrespectively, of whether the evaluation is federally (publicly) supported or funded by a nonprofit or philanthropy group. Evaluators must be ever cognizant of their duty to the public and recognize the public as an inescapable partner in their evaluation practice. This value is particularly reflected in two of the Guiding Principles for Evaluators, including Principle D: Respect for People, which states that "Evaluators respect the security, dignity and self-worth of respondents, program participants, clients, and other evaluation stakeholders," and Principle E: Responsibilities for General and Public Welfare, which states, "Evaluators articulate and take into account the diversity of general and public interests and values that may be related to the evaluation" (AEA, 2004, p. 2). Greene (2005) eloquently characterizes evaluation research as a "public craft" and evaluators as "stewards of the public good" (p. 7). Evaluation researchers can best work in the public interest and on behalf of African American communities by engaging their practice in a manner that produces high quality data that will: (a) generate a more profound understanding of the context of education for African American students, (b) help educators and relevant decision makers strengthen African American students and the schools that serve them, and (c) enlighten and empower African American families and communities to advocate for African American students.
Evaluation research on behalf of African American students and their communities must begin with a framework of designing and implementing evaluations to yield data that promote the creation of more equitable school reform that elevates the achievement of students. Those researchers entrusted with evaluating the educational outcomes of African American children must do more than simply analyze and report students' scores on standardized tests. In addition, evaluators must examine a host of social and other contextual factors (e.g., issues of access, equity, opportunities to learn, rigor of curricula) that are likely correlates of student outcomes to reveal a more complete picture. Examination of the interaction of the student culture with the culture of teaching, learning, and assessment allows evaluators and other professionals to develop more effective approaches to determine students' knowledge and ability (Hughes, 2001). Data obtained from a more comprehensive evaluation approach will yield information that moves beyond "blaming the victim" by providing invaluable information in which schools, parents, and communities can use to improve student learning. Rodriguez (2001) argues that evaluation researchers bear much of the responsibility for bringing about educational reform aimed at elevating the achievement of low-performing students instead of reporting data that continue to blame students. Often evaluators attribute the failure to deficiencies in the social, cultural, and linguistic experiences of the students' themselves, and tangentially, if at all, to the organization of learning and the resources committed to such learning organizations.
One avenue through which evaluations for the public good and on behalf of African American students can be greatly improved is increasing the capacity of the nation's educational evaluation enterprise and enhancing the presence and participation of more culturally attuned and minority evaluators. Within the past six years, the NSF has convened a series of evaluation workshops that included an invited cadre of nationally noted evaluation professionals who were charged with discussing issues related to evaluation training and practice, particularly in relation to evaluating educational programs targeting diverse and underserved populations. From these efforts, two sets of important proceedings were published: The Cultural Context of Educational Evaluation: The Role of Minority Evaluation Professionals (NSF, 2001) and The Cultural Context of Educational Evaluation: A Native American Perspective (NSF, 2003). These publications highlighted the need for more formal evaluation training programs, more highly trained evaluators, and an expanded view of evaluation education training to cover a very important aspect of program evaluationculture and context.
CONDUCTING QUALITY EVALUATIONS: PERSPECTIVES FROM THE HOWARD UNIVERSITY EVALUATION TRAINING INSTITUTE
Evaluators generally agree that there are three major phases in the evaluation research process: (a) Phase I-conceptualization and design of the evaluation, (b) Phase II-implementation, and (c) Phase III-reporting and disseminating results. To conduct educational evaluations on behalf of African American children and communities, it is essential that evaluations are planned and implemented in a manner that is both technically sound and culturally responsive. Stakeholder involvement, which has long been an expectation of good evaluation practice, is also a key element in this process. Providing stakeholders, especially those who traditionally have had less powerful roles in discussions of school reform, an opportunity to have a voice in the evaluation research process can help garner the needed trust and respect to guide the evaluators to the right questions which get the required information.
The Howard University Evaluation Training Institute (HU/ETI), an NSF-funded training grant, presents a conceptual framework and a set of steps for planning, implementing, and reporting evaluations. The HU/ETI framework is more detailed than the broad three-step approach described previously. It also articulates characteristics of good evaluators and evaluations from a perspective that is keenly attuned to the needs of African American and other communities of color. The HU/ETI conceptual framework characterizes good evaluators as individuals who have technical expertise, interpersonal skills, self-reflective abilities, and appropriate cultural and contextual knowledge of the programs and people under study. In keeping with the Program Evaluation Standards of the Joint Committee on Standards for Educational Evaluation (1994), the HU/ETI framework espouses that good evaluations are those that have utility, feasibility, propriety, and accuracy. Yet, adequately addressing cultural and contextual issues are essential to meeting these standards.
The HU/ETI framework envisions evaluators as individuals who are reflective practitioners, change agents, and culturally and contextually competent researchers. Being a reflective practitioner is meant to characterize someone who is engaged in an ongoing cycle of self-observation and thinking about the values, beliefs, and assumptions that guide his or her evaluation practice. As reflective practitioners, evaluators focus both internally on their own practices and externally on the social and contextual conditions of their practice. Evaluators should aspire to be agents for more equitable social, economic, and political transformation, contributing to the process of decision-making by the nature of their practice and the ways that they relate to stakeholders and decision makers. Culturally and contextually competent evaluators, as an integral part of the HU/ETI conceptual framework, refers to practitioners who possess a set of technical and interpersonal skills that allow them to increase and use their understanding and appreciation of cultural and contextual differences and similarities within, among, and between groups by having to draw on community values, traditions, and customs. Figure 1 provides a diagrammatic depiction of the HU/ETI conceptual framework.
The HU/ETI details the planning and conduct of evaluation research across five phases. Noticeably present in this framework is the ongoing stakeholder involvement, which is viewed as critical and bi-directional, during every phase of the evaluation process. While recognizing that each evaluation must be tailored to a specific set of contextual circumstances, the HU/ETI delineates five general phases of evaluation research: (a) Phase I: plan for the evaluation, (b) Phase II: design the evaluation, (c) Phase III: implement the evaluation, (d) Phase IV: analyze and synthesize findings, and (e) Phase V: disseminate findings for project improvement and use.
Phase I: Plan for the Evaluation
The first step in any evaluation study is to plan for the evaluation. This planning must take place long before the initiation of any data collection activities. Phase I essentially involves a thorough analysis of the evaluation context and the setting of boundaries related to the nature and scope of the research. During the planning stage, the evaluator must make deliberate decisions about what to study, who to study, what types of evidence is essential to meet the needs of the program and other key stakeholders (e.g., funders), and the resource requirements. Evaluators must carefully analyze the project's cultural and sociopolitical context as it presently exists in order to help establish the parameters of the research effort. This can be effectively achieved through the collection of background information and existing program constraints. In addition, the evaluator must obtain consensus about the purpose and goals of the evaluation as well as the appropriate evaluation questions to guide the research endeavor. The evaluator can substantially enhance his or her success in this effort through engaging key stakeholders, gaining their trust and cooperation, and facilitating their ownership of the evaluation.
During the planning phase, an evaluation team may need to be assembled. Evaluation studies which involve ethnically diverse participants and stakeholders often call for the creation of multiethnic evaluation teams to increase the chances of hearing the voices of underrepresented students (Frierson, Hood, & Hughes, 2002; Stevens, 2000). Members of the evaluation team should be respectful of the culture of the program and its participants as well as be competent to work in these diverse settings.
Phase II: Design the Evaluation
After the evaluator (or evaluation team) has achieved a good understanding of the program's context and achieved consensus regarding the evaluation's purpose and guiding questions, the design of the evaluation must be articulated. In recent years, there has been much discussion about the "best" design for evaluating educational programs. The Institute for Education Sciences (IES) of the U.S. Department of Education, in particular, purports that well-designed and implemented randomized controlled trials (RCTs) are the "gold standard" for evaluating an intervention's effectiveness, and it calls for use of RCTs in educational evaluations (U.S. Department of Education, 2003).
The debate regarding the use of RCTs in educational evaluations has received considerable attention in the evaluation community. This was especially in light of IES's commitment to privileging RCT designs over other methods in evaluation funding competitions. The AEA issued a response in October 2003 opposing the efforts to privilege RCTs in education funding competitions noting, among other things that: (a) RCTs are not always best for determining causality and can be misleading, (b) RCTs should sometimes be ruled out for ethical reasons, and (c) in some cases, data sources are insufficient for RCTs (AEA, 2003). Rossi, Lipsey, & Freeman (2004) argue that evaluators are all too often confronted with situations where it is difficult to implement the "very best" research design. Time and resource constraints always limit design options and that the justification for using the very best design, or the RCT gold standard, which is often the most costly design, varies with the importance of the intervention being tested and the intended use of the results (Rossi et al.).
The choice of research design inevitably involves trade-offs. As such, Rossi et al. (2004) advocate using the "good enough" rule in formulating a research design-the evaluator selecting the strongest possible design from a methodological standpoint after having taken into account the potential importance of the results, the practicality and feasibility of each design, and the probability that the design chosen will produce useful and credible results. The evaluation design does not have to be the most rigorous or elaborate, but instead one that is appropriate for the situation under study.
In addition to randomized control trials, evaluators can select from a multitude of quasi-experimental designs that do not involve randomly assigned intervention and control groups. These include, for example, constructing control groups via matching procedures, equating groups by statistical procedures, and using reflective controls or estimating program effects from data obtained on the target individuals at multiple points-in-time. Evaluators are also encouraged to consider qualitative and mixed-methods designs to illuminate program context, implementation, and outcomes.
During Phase II, the evaluator must also select, adapt, or develop the data collection instruments. Selection of the instruments for the evaluation is not a simple matter of locating a published measure on a particular variable of interest. Clearly, evaluators must use instruments that are valid, reliable, and appropriate for the target population. On the one hand, deciding to use an existing instrument involves a process of examining measures that are available and the ultimate selection of the best instrument in terms of the evaluation's purpose and the target population. On the other hand, if an instrument must be developed, the evaluator must subject the newly developed instrument to the standards of good instrument development espoused in the literature. A decision must also be made whether pilot or pre-testing of the instruments is necessary. The cadre of instruments and data collection procedures used in the evaluation should yield a variety of information. This is advantageous because it allows the researcher to present a more thorough view of the program from different perspectives.
After the evaluation design has been determined, the evaluator is in a position to prepare a realistic time schedule and specify the budgetary requirements. An appropriate time schedule includes a listing of all the major evaluation activities to be accomplished and the corresponding initiation and completion times for each proposed activity. While carefully planning and scheduling of evaluation activities will not guarantee that the absence of problems during the implementation, analysis, and reporting phases, it will certainly minimize the likelihood of unexpected events creating insurmountable barriers to the successful completion of the research. During this phase of the evaluation, it is also helpful to finalize a scope-of-work plan and share it with relevant stakeholders. Key elements in the written scope-of-work plan include: (a) a description of the evaluation questions to be addressed, (b) the data collection methods to be used, (c) the evaluator's and relevant others' tasks and responsibilities, (d) a listing of the reports that will be generated, and (e) the required budget to complete the work.
Phase III: Implement the Evaluation
At this point, the evaluator puts into effect the plans developed in Phases I and II. In this phase the evaluator focuses on the extent to which (a) the deadlines are met, (b) the design and sampling plan are properly implemented, and (c) the instruments and procedures are administered as intended. Stakeholders should be informed about all implementation efforts. To the extent possible, they should be recruited as partners in the implementation process assisting with participant recruitment and data collection efforts.
Phase IV: Analyze and Synthesize Findings
In this phase, the evaluator must ensure that data analytic techniques used are valid and well justified in light of the evaluation's purpose and objectives. In the synthesis of the data, interpretations must be accurate, culturally appropriate, and well grounded in the findings. Stakeholders should have an opportunity to review evaluation findings and provide input into the synthesis and interpretation of results. Frierson, Hood, and Hughes (2002) suggest the creation of review panels, principally comprising representatives from stakeholder groups (e.g., students, parents, school personnel, community members), to examine the evaluative findings gathered.
Phase V: Disseminate Findings for Project Improvement and Use
Early in the research process, the evaluator should have determined how research findings would be reported and disseminated. The creation and dissemination of evaluation findings that will be truthful, relevant, and useful to program staff and other key stakeholders in their efforts to elevate student outcomes is key.
Most funders require a written report at the end of each funding period. There may be requirements at the project or funding level for additional reports (e.g., quarterly reports, interim reports, oral communications). Reports should be written in a clear, simple, concise, straightforward style. Traditional written reports typically include the following information: (a) an executive summary, (b) background of the project evaluated, (c) purpose, intent, and design of the evaluation study, (d) results, (e) discussion, and (f) conclusions and recommendations for improvement. Evaluators should also consider creative and innovative approaches to reporting findings such as the use of stories, photographs, cartoon, videos, drama, and poetry. There are numerous documents available in the published literature that provide guidance on communicating evaluation findings (e.g., Grab, 2004; Torres, Preskill, & Pionket, 1996, 1997; W. K. Kellogg, 1998). Irrespective of which reporting and disseminating technique is used, each must accomplish several things. First and foremost, reports must provide useful information to the target audience/s. secondly, reports should be disseminated in time to be useful. Thirdly, reports should be crafted and presented in such a manner that they immediately catch the attention of the intended audience by providing compelling information. As in the previous phases of the evaluation, stakeholders should be engaged in this phase as well. They should be given multiple opportunities to suggest additional reporting and disseminating strategies (beyond what was agreed upon in Phase I) and to provide feedback on the report.
The Need for Cultural and Contextually Responsive Evaluation Research
Throughout much of the previous discussion, the authors stress the need to attend to cultural and contextual factors throughout the evaluation process. This perspective is embedded throughout the HU/ETI conceptual framework. In recent years, there has been considerable discussion within the evaluation community by some scholars about the importance of issues such as culture, context, pluralism, and inclusiveness in evaluation (e.g., Frierson, Hood, & Hughes, 2002; Guzman, 2003; Kirkhart, 1995; 2005; Mertens, 2003; Stanfield, 1999; Thomas & Stevens, 2004; Thompson-Robinson, Hopson, & SenGupta, 2004). Culture is a predominant force in shaping students' behaviors, values, and institutions. Context, a broader term, includes the combination of factors (including culture) accompanying the implementation and evaluation of a project that might influence its results (Thomas, 2004). Such factors include geographic location, timing, political and social climate, economic conditions, and other things going on at the same time as the project. Context is the totality of the environment in which the project takes place. If findings from educational evaluations are to serve public good and social justice functions, evaluators must recognize that to the extent that their results are culturally and contextually inappropriate, they are at risk of promoting the creation or perpetuation of stereotypes of underrepresented and socially oppressed groups (Guzman, 2003).
Evaluators often focus on methods-related issues in evaluation planning and implementation, while failing to reflect on how their own personal characteristics and culture can promote or hinder public good and social justice interests in evaluation. Evaluators' values, beliefs, and prejudices can and do influence a number of critical aspects of the research process such as (a) what questions an evaluator asks and ultimately not ask, (b) what an evaluator illuminates and ultimately minimizes, (c) what evaluation approach is used and ultimately not used, (d) what data are collected and ultimately overlooked, (e) how interpretations are made and whose interpretations are held in high or low esteem, (f) what conclusions are drawn and what conclusions are not considered, and (g) how results are presented and to whom such results are disseminated. A culturally and contextually responsive approach to evaluation represents the best strategy for conducting evaluative research in the public interest, especially for a more ethnically diverse public, and on behalf of African American children. Discussion of specific strategies and examples of conducting culturally and contextually responsive evaluations are beyond the scope of this article, but can be found elsewhere (see, for example, Frierson, Hood, & Hughes, 2002; Hood, Hopson, & Frierson, 2005; Johnson, 2005; Thomas & Stevens, 2004).
Evaluations for public good and on behalf of African American students are not done to the students, the school context, and educational projects; but, instead, they are done for the students, the school, and the educational efforts under study. Conducting educational evaluation for public good and on behalf of African American students is indeed a noble research endeavor. Educational evaluators working in this area should seek to make a positive difference in the lives of African American students and the settings where evaluators work, changing them in some important and constructive ways through the data that are collected and the utility of the findings that are reported. Evaluations driven by this perspective are more likely to provide K-12 administrators and others with the data they need to make informed decisions about best practices for educating African American students.
American Evaluation Association. (2003). Scientifically based evaluation methods. Fairhaven, MA: Author. (The full position statement of the AEA is available at http://www.eval.org/ doepage.htm.)
American Evaluation Association. (2004). Guiding principles for evaluators (Rev. ed.). Fairhaven, MA: Author. (Full text available online at www.eval.org/Publications/Guiding Principles.asp)
Cooley, W. W., & Lohnes, P. R. (1976). Evaluation research in education. NY: Irvington Publishers.
Cronbach, L. J., Ambron, S. R., Dombusch, S. M., Hess, R. D., Hornik, R. C., Phillips, D. C., Walker, D. F., & Weiner, S. S. (1980). Toward reform of program evaluation. San Francisco: Jossey-Bass.
Elementary and Secondary Education Act of 1965, Pub. L. No. 89-10, 20 U.S.C. § 6301 et seq. (1965).
Frierson, H. T., Hood, S., & Hughes, G. B. (2002). Strategies that address culturally responsive evaluation. The 2002 user-friendly handbook for project evaluation (REC 99-12175). Arlington, VA: National Science Foundation, Directorate for Education and Human Resources.
Greene, J. C. (2005). Evaluators as stewards of the public good. In S. Hood, R. Hopson, &, H. Frierson (Eds.), The role of culture and cultural context (pp. 7-20). Greenwich, CT: Information Age.
Greene, J. P. & Winters, M. A. (2005). Public high school graduation and college readiness rates: 1991-2002. (Education Working Paper, No. 6.) New York: Manhattan Institute for Policy Research.
Grob, G. F. (2004). Writing for impact. In J. S. Wholey, H. P. Hatry, & K. E. Newcomer (Eds.), Handbook of practical program evaluation (2nd ed., pp. 604-627). San Francisco: Jossey-Bass.
Guzman, B. L. (2003). Examining the role of cultural competency in program evaluation: Visions for new millennium evaluators. In S. I. Donaldson & M. Scriven (Eds.), Evaluating social programs and problems: Visions for the new millennium (pp. 167-181). Mahwah, NJ: Erlbaum.
Harvard Family Research Project. (2002). A conversation with Michael Quinn Patton [Electronic version]. The Evaluation Exchange, VIII, 10-11.
Hood, S., Hopson, R., & Frierson, H. (Eds.) (2005). The role of culture and cultural context: A mandate for inclusion, the discovery of truth, and understanding in evaluative theory and practice. Greenwich, CT: Information Age.
Hughes, G. B. (2001). Evaluation of educational achievement of underrepresented minorities: Assessing correlates of student achievement. In The cultural context of educational evaluation: The role of minority evaluation professionals (NSF 01-431). Arlington, VA: National Science Foundation, Directorate for Education and Human Resources.
Johnson, E. (2005). The use of contextually relevant evaluation practices with programs designed to increase participation of minorities in science, technology, engineering, and mathematics (STEM) education. In S. Hood, R. Hopson, & H. Frierson (Eds.), The role of culture and cultural context: A mandate for inclusion, the discovery of truth, and understanding in evaluative theory and practice, (pp. 213-235). Greenwich, CT: Information Age.
Joint Committee on Standards for Educational Evaluation. (1988). The personnel evaluations standards: How to assess systems for evaluating educators. Newbury Park, CA: Sage.
Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards (2nd ed.). Newbury Park, CA: Sage.
Joint Committee on Standards for Educational Evaluation. (2003). The student evaluation standards: How to improve evaluation of students. Newbury Park, CA: Corwin Press.
Kirkhart, K. E. (1995). Seeking multicultural validity: A postcard from the road. Evaluation Practice, 16, 1-12.
Kirkhart, K. E. (2005). Through a cultural lens: Reflections on validity and theory in evaluation. In S. Hood, R. Hopson, & H. Frierson (Eds.), The role of culture and cultural context; A mandate for inclusion, the discovery of truth, and understanding in evaluative theory and practice (pp. 21-38). Greenwich, CT: Information Age.
Laird, J., Law, B., DeBell, M., & Chapman, C. (2006). Dropout rates in the United States: 2002 and 2003 (NCES 2006-062). Washington, DC: U.S. Department of Education, National Center for Education Statistics.
Mertens, D. M. (2003). The inclusive view of evaluation: Visions for the new millennium. In S. I. Donaldson & M. Scriven (Eds.), Evaluating social programs and problems: Visions for the new millennium (pp. 91-107). Mahwah, NJ: Erlbaum.
National Science Foundation. (2001). The cultural context of educational evaluation: The role of minority evaluation professionals [Workshop proceedings], NSF 01-43. Arlington, VA: NSF, Directorate for Education and Human Resources.
National Science Foundation. (2003). The cultural context of educational evaluation: A Native American perspective [Workshop proceedings], NSF-03-032. Arlington, VA: NSF, Directorate for Education and Human Resources.
No Child Left Behind Act of 2001. (2002). Pub. L. No. 107-110, 115 Stat 1425.
Olson, D., & Viadero, D. (2002, January 30). Law mandates scientific base for research. Education Week, p. 1.
Rallis, S. F., & Rossman, G. B. (2003). Mixed methods in evaluation contexts: A pragmatic framework. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social & behavioral research (pp. 491-512). Thousand Oaks, CA: Sage.
Rodriguez, C. M. (2001). Assessing underrepresented science and mathematics students: Issues and myths. In The cultural context of educational evaluation: The role of minority evaluation professionals (pp. 16-23, NSF-01-43). Arlington, VA: National Science Foundation, Directorate for Education and Human Resources.
Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A systematic approach (7th ed.). Newbury Park, CA: Sage.
Shadish, W. R. , Jr., Cook, T. D., & Leviton, L. C. (1991). Foundations of program evaluation: Theories of practice. Newbury Park, CA: Sage.
Stanfield, J. H. (1999). Slipping through the front door! Relevant social science evaluation in the people of color century. American Journal of Evaluation, 20, 415-431.
Stevens, F. I. (2000). Reflections and interviews: Information collected about training minority evaluators of math and science projects. In The cultural context of educational evaluation: The role of minority evaluation professionals (pp. 41-55, NSF-01-43). Arlington, VA: National Science Foundation, Directorate fro Education and Human Resources.
Teddlie, C., & Tashakkori, A. (2003). Major issues and controversies in the use of mixed methods in the social and behavioral sciences. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social & behavioral research (pp. 3-50). Thousand Oaks, CA: Sage.
Thomas, V. G. (2004). Building a contextually responsive evaluation framework. In V. G. Thomas & F. I. Stevens (Eds.), Co-constructing a contextually responsive evaluation framework: The Talent Development Model of School Reform. New Directions for Evaluation, No. 101 (pp. 324). San Francisco: Jossey Bass.
Thomas, V. G., & Stevens, F. I. (Eds.). (2004). Co-constructing a contextually responsive evaluation framework: The Talent Development Model of School Reform. New Directions for Evaluation, No. 101. San Francisco: Jossey-Bass.
Thompson-Robinson, M., Hopson, R., & SenGupta, S. (Eds.). (2004). In search of cultural competence in evaluation: Toward principles and practices. New Directions for Evaluation, No. 103. San Francisco: Jossey-Bass.
Torres, R. T., Preskill, H., & Pionket, M. E. (1997). Communicating and reporting: Practices and concerns of internal and external evaluators. Evaluation Practice, 18, 105-125.
Torres, R. T., Preskill, H., & Pionket, M. E. (Eds.). (1996). Evaluation strategies for communicating and reporting: Enhancing learning in organizations. Thousand Oaks, CA: Sage.
U.S. Department of Education. (2003). Identifying and implementing educational practices supported by rigorous evidence: A user friendly guide. Washington, DC: Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance.
Waters, G. A. (1998). Critical evaluation for education reform. Education Policy Analysis Archives, 6, 1-39. (Available online at http://epaa.asu.edu/epaa/v6n20.html.)
W. K. Kellogg Foundation. (1998). The evaluation handbook. (Available at the W. K. Kellogg Foundation's website at http://www.wkkf.org.)
Veronica G. Thomas Howard University
Brooke K. McKie Howard University
VERONICA G. THOMAS is Professor, Department of Human Development and Psychoeducational Studies, School of Education, Howard University, Washington, DC.
BROOKE K. MCKIE is Project Director, Center for Urban Progress, Howard University, Washington, DC.
All comments and queries regarding this article should be addressed to firstname.lastname@example.org…