Kirkpatrick's Level 1 Evaluation of the Implementation of a Computer-Aided Process Design Tool in a Senior-Level Engineering Course
Diefes-Dux, Heidi A., Samant, Charuta, Johnson, Tristan E., O'Connor, Debra, Journal of Engineering Education
Computer simulation tools are frequently used in engineering design work, and undergraduates are often trained to use these tools as they learn to design systems. The use of new tools in the learning environment should be evaluated to assure that the students are able to use the tools effectively. This study details and demonstrates the use of a Kirkpatrick's Level 1 Evaluation to assess the effectiveness of an instructional environment in which students learn to use a computer simulation tool to perform engineering design work. Specifically, an evaluation was conducted to look at student perceptions of FOODS-LIB-a steady-state food process design tool, its user's manual learning modules, and the implementation of FOODS-LIB in a senior level design course. This evaluation was triangulated with an instructor's assessment of student products generated as the students used the learning modules and designed an ice cream manufacturing process using FOODS-LIB.
Keywords: Kirkpatrick, evaluation, design tool
"The term 'computer simulation' is used to describe a computer program which incorporates a mathematical or logical model of an engineering system or process, allowing the user to specify the values of one or more system parameters and, following computation, to examine the resulting values of one or more system parameters" . Regardless of engineering discipline, computer simulation tools are increasingly used in engineering practice to facilitate various aspects of design work. Take for instance process design, which encompasses a broad array of activities from process conceptualization to detailed process design. Flowsheeting, the performance of steady-state calculations necessary to describe the behavior of a process and to determine key operating conditions , is just one step in the design of any process. It is, however, the most labor-intensive activity for any process engineer. Computer aided process design tools are increasingly recognized as an essential component of any engineer's toolkit as their use reduces computation time and allows for more aggressive parametric analyses.
It has been recognized for over twenty years that universities have a responsibility to ensure that their graduates are prepared to use or learn to use these tools . This notion is reinforced by ABET's Engineering Criterion 3: Program Outcomes and Assessment which states that "Engineering programs must demonstrate that their graduates have: [...] (c) an ability to design a system, component, or process to meet desired needs [and] (k) an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice" . However, effective incorporation of computer simulation tools in an engineering curriculum is often difficult because these tools tend to have a steep learning curve and often require some programming. The tendency is to give little consideration to the process by which students learn to use these tools effectively as compared to their learning engineering science concepts and applying these concepts in design situations.
There is undoubtedly an instructional design dilemma. Students' ability to use a tool is not, in and of itself, useful if students do not have a fundamental understanding of and ability to apply engineering science and design principles. That is the equivalent of knowing how to use a word processor but having no command of the English language or the subject matter. On the other hand, students cannot acquire good engineering design sense (e.g., feel for size and sensitivities) without the use of simulation tools to aid in design activities that are resource intensive (e.g., time and cost).
Can learning modules and implementation protocols be developed to effectively facilitate students' learning to use features of the tools and transferring their knowledge to actual engineering design situations? While many different engineering design tools and associated learning modules have been created and implemented in higher education settings, there is limited literature concerning the evaluation of these computer simulation tools, their implementation, and the nature and quality of their subsequent use by students in their design work. …