Using Evaluation to Adapt Health Information Outreach to the Complex Environments of Community-Based Organizations*[dagger]
Olney, Cynthia A., Journal of the Medical Library Association
Objective: After arguing that most community-based organizations (CBOs) function as complex adaptive systems, this white paper describes the evaluation goals, questions, indicators, and methods most important at different stages of community-based health information outreach.
Main Points: This paper presents the basic characteristics of complex adaptive systems and argues that the typical CBO can be considered this type of system. It then presents evaluation as a tool for helping outreach teams adapt their outreach efforts to the CBO environment and thus maximize success. Finally, it describes the goals, questions, indicators, and methods most important or helpful at each stage of evaluation (community assessment, needs assessment and planning, process evaluation, and outcomes assessment).
Literature: Literature from complex adaptive systems as applied to health care, business, and evaluation settings is presented. Evaluation models and applications, particularly those based on participatory approaches, are presented as methods for maximizing the effectiveness of evaluation in dynamic CBO environments.
Conclusion: If one accepts that CBOs function as complex adaptive systems-characterized by dynamic relationships among many agents, influences, and forces-then effective evaluation at the stages of community assessment, needs assessment and planning, process evaluation, and outcomes assessment is critical to outreach success.
For several months, an outreach team from a South Texas medical school library carefully planned a community outreach training session and focus group to be held at a community center serving a low-income Hispanic colonia (community) in the Lower Rio Grande Valley. The outreach team developed relationships with community center staff and engaged their assistance in the planning. The team offered preliminary training to residents of the community, generating enthusiasm and engaging some of them as outreach partners. These partners, in turn, personally invited other community members to the training session and focus group. The team purchased gift cards from a local grocery store as incentives to attract participants and hired an experienced focus group facilitator from South Texas to conduct the session in Spanish.
The final session was scheduled for ten o'clock on a weekday morning in mid-March. When the outreach team arrived at the site, community center staff warned that attendance probably would be very low because of bad weather: the temperature had dipped to thirty-seven degrees. Outreach team members, particularly those who had grown up in middle-class neighborhoods in northern climates, could not imagine how plans could fall apart over such weather-a dry day, with temperatures above freezing. However, the community staff predictions were accurate. Perhaps because the families lacked adequate clothing or household heat, this weather was bad enough to keep the children home from school and mothers from attending the training session. It was, in fact, a slow day for outreach.
Most would call this an example of Murphy's Law: what can go wrong, will go wrong. It also demonstrated the unpredictable nature of the environments of community-based organizations (CBOs). Funding sources come and go. The turnover of low-paid and volunteer workers tends to be high. Changing politics lead to changing priorities. The involvement of and relationship between stakeholders can make decisions and progress difficult. For instance, Scherrer outlined the challenges faced in an environmental public health outreach project in Chicago: lack of institutional support of Internet technology, participants unwilling to show lack of knowledge until they knew and developed trust in the outreach coordinator, difficulty in building cooperative relationships among volunteers, and unique culture of each organization .
Such challenges make outreach, as well as evaluation, quite difficult. …