Designing Exercises for Teaching and Analysis: Center for Applied Strategic Learning
Exercises are designed for purposes that can generally be collapsed into two overarching goals: teaching or analysis. The goal of teaching is usually to make theoretical lessons concrete and convey some aspect of the demands that a student might face in applying them. When we use an exercise as an analytical tool, in contrast, we use it as a model that represents some real world problem or, better, class of problems and uses participant actions to generate information about how at least one of the elements of that model impacts decisionmaking. In this article, we discuss design process and examine the ways in which exercise purpose impacts its form, particularly its scale. Perhaps controversially, we also cast doubt on the analytical utility of large-scale exercises.
Games successfully used for teaching purposes appear to incorporate a number of factors. They are rich and detailed enough to excite and compel participants. They have many different functional roles for participants, giving them some representation of the experience of performing those duties, the more realistic the better. They accurately convey the complexity of the real world and require them to make responses to sudden developments, the more unexpected the better. The lessons that participants learn and are to apply to the real world have more to do with process than outcome and often simply underscore the difficulty of making choices in the thick of things.
The more specific and detailed the scenario or exercise, however, the more limited the conclusions that can be extrapolated from it to other problems or situations. If we are conducting an exercise to explore the contours of some ill-defined future problem, for instance, it is crucial that we be able to justify why we reach certain conclusions or how we generalize lessons learned from an exercise. Answering the "How do I know that I know that?" question is routine in the social sciences, including in qualitative work common in political science and sociology, but not always thoroughly discussed in the exercise design and evaluation community. Nevertheless, it is crucial to a defensible analysis.
An exercise that will be the basis of or contribute to an analytical study needs to incorporate features that allow investigators to generalize some findings and explain why their conclusions are not contingent on a random scenario detail or quirk of a particular participant. Here, then, parsimony trumps detail, and we are more interested in the smallest number of shared factors that might be causally related to outcomes and solutions to a problem. There is a variety of interesting work on the ways in which qualitatively specified games can be used analytically, ranging from being bundled together to validate formal mathematical models to serving as mechanisms for aggregating the expert knowledge of participants.
The elements of good exercise design for teaching and analysis can be somewhat different for the simple reason that the lessons to be learned are different. Analytically, what we learn from tabletop exercises usually has to do with whether the model of the problem described in the scenario introduces the right independent variables, whether others should be added, how they could be refined and their relative weight, and how differences in them might require different actions and result in different outcomes.
Exercises for teaching purposes are rooted in an assumption of the value of experiential learning, that giving participants a visceral feel for the exigencies of policy decisionmaking will be an effective way of making theoretical lessons they have learned concrete. For this reason, exercises are frequently used as capstones to courses, particularly at U.S. graduate military education institutes, and a single iteration of them more than suffices for teaching purposes, though problematic for an analytical exercise. …