Clicking to Learn: A Case Study of Embedding Radio-Frequency Based Clickers in an Introductory Management Information Systems Course
Nelson, Matthew L., Hauck, Roslin V., Journal of Information Systems Education
The challenges associated with teaching core introductory management information systems (MIS) courses are shared by many. After acclimating to the sheer size of the lecture room and the volume of students (often with triple digit enrollments), the instructor must confront the larger issue of actively engaging students and sustaining their interest throughout the semester. This problem is exacerbated with the reality that a majority of students in an introductory MIS course are non-MIS majors, whom have little motivation to attend or to actively participate. This large lecture format also imposes common problems that are difficult to overcome with traditional teaching practices. Taking attendance is nearly impossible, seating charts are out of the question, in-class student activities are a logistical nightmare, group projects are too time consuming to properly manage, in-class surveys are great (but who has time to count all of the hands) and the list goes on.
This paper is intended to address these challenges by presenting a real life application of an in-class, radio-frequency (RF) based classroom response system (CRS; a.k.a. "clickers") in a core introductory MIS course for business students. Although the use of clickers in a classroom is no panacea, they most certainly provide significant inroads towards managing the challenges inherent in a large lecture format course. The driving research questions motivating this study include: how can CRS be effectively used to assist with overcoming inherent challenges of large lecture classrooms? What system features and traits should be sought in a classroom response system? Using a background of research in effective teaching and learning, this case study details the technical implementation and integration of an RF-based classroom response system with the course curriculum, as well as the significant benefits, lessons learned and effective practices discovered in the process. This manuscript is vendor-neutral, with a focus on system traits, features and practices that were found to be effective in enhancing the teaching and learning process.
2.1 A Review of Effective Learning Principles in the Classroom
Before discussing the technology, one should first consider the underlying goals to be accomplished by implementing CRS in the classroom. The research literature on learning discusses a number of concepts that promote effective learning in the classroom. Active learning, providing feedback, increasing attention span, and motivation are the four learning principles that have been identified as being particularly challenging to the large-lecture format class (Beatty, 2004, Bergtrom, 2006) and of Net Generation learners (Robinson and Ritzko, 2006). Active learning refers to techniques that require students to actively process and apply information to learn as opposed to passive listening (Meyer and Jones, 1993). The key characteristic of active learning that makes it a powerful technique is that active engagement involves higher-order thinking (such as analysis, synthesis, evaluation), which results in greater learning, understanding, and retention (Beatty, 2004; Bonk and Cunningham, 1998; Bonwell and Eison, 1991; Thalheimer, 2003).
Providing feedback in the classroom is another technique that has been found to enhance learning (Bangert-Downs, et al., 1991). Successful feedback in the classroom refers to providing information to the student that draws attention to the learning process, thereby improving performance in the classroom (Kluger and DeNisi, 1996). Not only is feedback important, the timing in which the feedback is given is also important. A number of studies have found that immediate feedback is more effective than delayed feedback (see Azevedo and Bernard, 1995; Kulik and Kulik, 1988). The process of giving feedback repeatedly can also help maintain student attention span.
Attention span deals with selectively attending to and extracting information from the environment (Bandura, 1986). In the learning environment, the purpose of attention span is to keep students actively processing information that is relevant to the class (Anderson, 1995). Because research on attention span and learning has indicated that student attention spans average ten to eighteen minutes (Johnstone and Percival, 1976), the ability to continuously reengage students during a lecture class is an important technique to enhancing learning.
Closely related to all three of the learning concepts discussed is student motivation. A number of previous research studies have found that motivation can play a key role in improving performance in the classroom (see Ames and Archer, 1988; Weiner, 1990). Students who are motivated to learn do actually learn more than students who are not motivated (Frase, 1971). In addition to using active learning and providing feedback to increase motivation, research indicates that reducing anonymity in the classroom also provides motivation for students (Sorcinelli, 1991). Given the challenges of teaching an introductory large-lecture MIS course, it became evident that in order to incorporate effective learning principles, an evaluation of technological innovations that support our teaching goals and seek to improve classroom performance was necessary.
2.2 Classroom Response System (CRS) Overview
The innovation evaluated in this study is a radio-frequency (RF) based classroom response system (CRS), or simply referred to as "clickers". A close analogy that one can use to describe clickers is with that of a remote control device for televisions (TV). Beatty (2004) provides an excellent primer on this topic covering Classroom Communication Systems. Similar to TV remote controls, clickers are wireless, handheld devices that have fewer buttons to push than a TV remote control. Most clickers have a 10-key alpha / numeric keypad, with two extra buttons for on / off and changing RF channels. Clickers are much smaller than TV remote controls, with most clickers no larger than a standard credit card and about three times thicker. Another important difference between TV remote controls and clickers is the type of wireless communications protocol used in the device. While most TV remote controls use infrared (IR), many clickers now make use a radio-frequency (RF) based wireless transmission. Compared with radio-frequency, IR transmission is less expensive, but is older and considered a less reliable transmission since it is a line-of-sight protocol (e.g. nothing can be physically blocking the sending and receiving devices). Much like a car radio, RF is a radio-frequency based wireless transmission and is not encumbered by most physical obstructions. As such, RF can be used over longer distances and can transmit signals much faster (between the sending and receiving devices) than IR-based devices. Although clickers have been used in classroom settings since 1985 (Beatty, 2004), the use of RF in clickers is a recent development that began in earnest in 2004. In fact, prior to the use of RF, the predominant transmission medium for clickers was IR.
In a classroom setting, a formal CRS traditionally consists of a RF receiver device (a.k.a. dongle), application software, and handheld devices (a.k.a. clickers) that students use to input responses. Dongles are installed directly in a universal serial bus (USB) slot in the instructor's PC station of a classroom. The software application can be served or (as typically the case) loaded directly on the instructor's PC station. Each student is assigned, or has purchased, a clicker that possesses a unique RF identifier (RFID). Because each clicker uniquely identifies its owner, instructors can use the data collected by the system to easily track student participation and performance. Thus, for example, an instructor would use the CRS application software to present multiple choice questions during their lecture. If five answer choices are provided (A through E) with each multiple choice question, students would then enter their answer choice in their clicker. The clicker sends an RF signal (that possesses the student's unique RFID and her answer selection), which is received by the RF dongle and electronically stored by the application software in the instructor's PC station. The software will automatically and instantly store, grade, tally, and report scores for every each student, for 'groups' of students, for the entire class, for each lecture, for semester to date, and so on.
3. A CASE STUDY OF IMPLEMENTING CRS IN AN INTRODUCTORY MIS COURSE
A case study was chosen as the assessment approach utilized in this study. Case study assessments are appropriate when a researcher is seeking greater depth in understanding of an emerging phenomenon and when examining "how" and "why" types of research questions (Alavi, Yoo and Vogel, 1997; Leidner and Jarvenpaa, 1993). The driving research questions in this study include examining how a CRS can be effectively used to assist with overcoming the inherent challenges of a large lecture course? What system traits and features should be sought in a classroom response system? To fully address these questions, the case study is structured consistent with phases in the traditional systems development life-cycle (SDLC), by walking through systems analysis, pilot testing, design modifications, implementation and integration with the course curriculum. The subsequent discussion emphasizes lessons learned, effective practices and teaching tips gained from the entire experience.
3.1 Research Setting
The case setting is a medium-sized public university located in the mid-western United States with a student enrollment averaging 20,000 students and a College of Business (COB) enrollment averaging 3,200 students. During a 16-month time frame (from fall 2004 through the end of fall 2005 semester), the authors collaborated and coordinated efforts between software vendors, publisher representatives, the university network support groups (at multiple levels), the classroom technology support unit, the university bookstores, and other faculty to develop a manageable and compatible clicker solution. The CRS solution developed as a result of this study was subsequently approved and adopted as the university-wide clicker solution in December 2005.
3.2 System Development and Implementation
Requirements determination and initial system design work began in September 2004. Although many software vendors offer packaged off-the-shelf clicker solutions (most of which were evaluated for purposes of this study), the initial focus was capturing requirements from stakeholders (faculty, students, technical staff and publishers) and insuring compatibility with the university's network and classroom technology. This phase was particularly challenging since the college moved into a newly constructed facility in January 2005. A pilot system was tested in a small MIS course of 31 students from February through April 2005. Although student feedback was overwhelmingly favorable, the pilot system was technically unstable. In fact, the pilot system worked only 33 percent of the time. The two primary causes of failure were traced back to the use of antiquated IR based clickers and numerous problems associated with the use of an off-campus system administrator that was provided by a third party software vendor.
The pilot resulted in several changes to the system, including switching to on-campus system administration (as opposed to an outsourced administrator), using radio-frequency based clickers (as opposed to infrared clickers), and utilizing an CRS application program that is highly integrated with Microsoft PowerPoint[R] (as opposed to a stand-alone independent application). Collectively, these changes brought dramatic improvements to the system. For example, all of the non-value added complexities of working with an outsourced administrator were eliminated (e.g., waiting for the off-site administrator's data to refresh, maintaining an uninterrupted network connection, requiring students to share private information with an outside organization, and registering the course, the instructor and the students with the outside organization each semester). Although this shifted some of the system administration burden back on the instructor, it was determined that the greater control, the enhanced student privacy, and the elimination of non-value added activities far outweighed the burden. In addition, the use of RF-based clickers enabled the clicker-to-receiver hit ratio to dramatically rise from the 40 ~ 45% range (with infrared) to the 93 ~ 98% range (with RF). Since infrared technology is an older line-of-sight protocol, students were often forced to stand, physically point and repetitively click towards the front of the classroom in hopes that the dongle would successfully received their response. The use of radio-frequency (RF) based clickers dramatically improved this clicker-to-receiver hit ratio and virtually eliminated these problems. Finally, switching to an application that is integrated with Microsoft PowerPoint[R] made the system more user-friendly and easier to integrate into class lectures. Rather than learning an entirely new stand-alone clicker application (as was the case in the pilot test), the new system acts as an add-in program in PowerPoint[R], giving the application a consistent look, feel and menu structure. This reduced the instructor's (and students) learning curve with the new system.
3.3 Integration with Course Curriculum
The updated CRS was implemented in four introductory MIS courses during the fall 2005 and spring 2006 semesters (totaling 588 students). To integrate the technology into the classroom, instructors utilized a number of learning activities that support the four concepts of effective learning previously discussed (i.e., active learning, feedback, attention span, and motivation).
3.3.1 Active learning: Lectures were enhanced to include questions pertaining to the covered material. Instructors were able to electronically take attendance, administrator in-class quizzes, conduct surveys and integrate student questions during lectures. Students were provided an enriched active learning environment via electronic material comprehension "checks", practice exam questions, opportunities to compare their performance with that of their peers, opportunities to earn credit for active classroom participation (including regular class attendance) and many others.
3.3.2 Providing feedback: CRS was also used to provide instantaneous feedback regarding the correct response and the distribution of responses among possible answers (for the entire class). The CRS lecture questions enabled the instructor to instantly gauge learning (for each student and the entire class) and enables students to instantly gauge their own comprehension and in relation to the entire class. The proportion of course points attributable to CRS usage is four to six percent for class participation and two to three percent for class attendance. Prior to CRS, both proportions were zero percent since there was no reliable and accurate means to automatically track student participation and attendance. Attendance is graded as a dichotomous measure dependent upon whether the student is in class or not. Participation grading generally does not penalize students for incorrect responses, but students do lose points for failing to participate. Students were informed that the CRS questions presented during lectures are similar to those that they can expect to see on exams. Some exceptions to this general rule include unannounced pop-quizzes, in-class student break-out sessions and in-class demonstrations / activities. Various student opinion surveys, prior exam questions and guest speaker Q&As have also been integrated in the classroom using CRS. Similar to lecture questions, these items are prepared in CRS during normal course prep (prior to the start of class) and the results are tallied, stored and reported back to the students in a matter of seconds.
3.3.3 Attention Span: During lectures, CRS was used extensively to conclude a topic (with summary questions), to change the pace of classes (with randomly timed student opinion surveys), and to ease transitions to a different topic (rhetorical questions to spark interest). Designing lectures so that students participate in clicker sessions two to three times per class not only requires that each student actively participate and think about the lecture material, it also serves as a way to change the rhythm of the class and prevent the lecture from becoming stagnant.
3.3.4 Motivation: The system also provides a means of motivating students to participate and be more interested in the class material. For example, using the system to collect and display results from thought-provoking, in-class opinion polls gives the instructor a means of incorporating student feedback into class discussions. Although incorrect answers were generally not penalized, students knew their responses were not anonymous and instructors still had the capability to evaluate how they were performing in each class. To further motivate interest, consolidated results are often subtotaled for students along various demographics (e.g., male versus females, juniors versus seniors, Accounting majors versus MIS majors, etc.). Students can also be assigned to or choose a team and the system tabulates average correct scores for each team. This team dimension approach is consistent with The Question Cycle--An Effective Model for [Classroom Communication Systems] Use in Class, that can "dramatically transform the classroom environment and entire learning dynamic for a course ... that instructors and universities are not accustomed to" (Beatty, 2004, p5). Inclusive in this is a different type of motivation that students enjoy (with friendly competition) and the challenge of the activity.
3.4 Student Feedback and Performance Results
To supplement the initial findings and to collect student perceptions and feedback regarding CRS, surveys were conducted in three sections of the same course during the spring 2006 semester. The surveys were administered anonymously during the second to the last week of a 16-week semester course, when students knew 65% of their overall course grade immediately prior to completing the survey. Although the three sections had different instructors, the title and purpose of each course was the same (Introduction to Management Information Systems) with each instructor utilizing the same textbook and similar evaluation materials. For comparison purposes, Section A is considered high-end usage (with 270 clicker questions during the semester), Section B is considered low usage (with 50 clicker questions during the semester), and Section C is a non-adopting control group (no clicker usage during the semester). Survey items were measured on a 5-point Likert scale ranging from Strongly Disagree (1) to Strongly Agree (5). Table 1 provides survey items and results that were administered across all three sections, including comparisons of student performance, class attendance and attendance / performance correlations. Table 2 provides additional survey items and results that were administered exclusively to actual student users of clickers (Sections A and B only). Both tables provide significance levels of standard statistical two tailed t-tests conducted for the difference in means in survey items, between the sections.
As summarized in Table 1, students' preference for the use of clickers in a classroom setting does significantly increase from non-adopters through the high-usage sections, as does the students' perception towards the use of clickers being able to improve their overall class performance. Student attendance significantly increased from the non-adopter section up through the high-usage section as well. Finally, in terms of actual student performance, mean student grades did not necessarily improve with greater levels of clicker usage. However, the correlation between student attendance and actual grade did significantly improve from the non-adopter section up through the high-usage section.
Furthermore, a comparison between the High Usage group (section A) and Low Usage group (section B) in Table 2 reveals that section A students evaluated the clicker technology more favorably than that of Section B students across all survey items. (In fact, eight of the mean score item differences are statistically significant at the p<.01 level or lower). Interestingly, the most significant improvements areas from Section B to Section A occurred in core Active Learning and Motivational related survey items ("The use of clickers helps me prepare better for exams", "I learn more as a result of using the clickers" and "I remembered lecture material more as result of clickers"). The least improvement item between Section A and B pertained to the technology itself, where volume (usage) level variations are less likely to impact student perceptions (the instructor's PC station and CRS have precisely the same build). Collectively, these findings are certainty encouraging for advocates of clickers in the classroom and provide promising results towards their effectiveness in helping to overcome the inherent challenges of a large lecture courses. The results in Table 2 also introduces the notion that the greater the usage of clickers during a course, the greater the student's perceived improvements towards active learning, motivation and feedback related survey items. The reader should be cautioned of the limitations of this approach. Although the researchers did seek to control for many variables through the structure of the survey (same course, same textbook, same semester, similar student profiles), there are other factors that influence these results that were not controlled for or measured. Some of these include variations in instructor teaching styles, time of day, number of examinations, number of supplemental assignments, and many others. Thus, as stated at the outset, the survey results supplement the real focus of this research which is the case study, including the findings associated with the systems development effort, the integration with the course curriculum (and the classroom) and the lessons learned and effective practices provided in the discussion below.
The discussion is structured along three lines: the system benefits; lessons learned; and effective teaching practices discovered throughout the development, implementation and use of CRS. The discussion is supplemented with quantitative and qualitative student survey results. The lessons learned are a summary of the recommendations based on the authors' experiences, both positive and negative. The effective practices are a summary of the techniques and other instructional strategies used in the integration of CRS into an introductory MIS course curriculum. Our hope is that this case study and reflective discussion will provide essential insights into the successful development and effective integration of CRS for current and future adopters. See Figure 1 for a summary of findings.
4.1 System Benefits
The student surveys provided excellent insights into the benefits enabled by CRS. The following student benefits can be elicited from the survey results and are based on a 5-point Likert scale ranging from Strongly Disagree (1) to Strongly Agree (5) from the combined results of 240 students from both adopter sections A and B (see Table 2). In descending order, students felt that the use of clickers encouraged them to participate more during classes (4.23) and to attend more classes (3.99). Students also liked the ability to see how other students answered questions (3.96) and they felt the use of clickers made the class more interesting (3.94) and improved their ability to remember material covered in lectures (3.91). In fact, students felt the use of clickers helped them to better prepare for exams (3.78) and, on an overall basis, they learned more (3.76) and performed better in the class (3.66) as a result of using clickers.
Based on open-ended survey questions asking students to provide their opinion on the CRS technology, students found the system useful for self-assessment and for comparing their performance against that of the entire class. The instant feedback and the ability for instructors to use the feedback to elaborate on a topic was also a benefit that students reported. The use of the system to break up the class and add interactivity enhanced attention span throughout the class period. Similar to the class enjoyment and student participation desires found in Robinson and Ritzko's study (2006), our respondents also reported that the use of the technology "lightens up the classroom experience" and "feels like a game show when the audience buzzes in for answers."
From an instructor's point of view, the data collected from the classroom response system is a valuable and efficient means towards monitoring student performance. The instant feedback provides instructors the opportunity to adapt their teaching in respond to the class's immediate needs. It also enables instructors to easily and continuously monitor class and individual student performance. It provides instructors an opportunity to identify students who may be struggling before (rather than after) examinations.
4.2 Lessons Learned
The following are key lessons from the systems development, implementation and integration with course curriculum:
4.2.1 Avoid using an infrared-based system: In addition to the low student clicker-to-receiver hit ratio (which only averaged 40 to 45 percent in the pilot study), the technology is often plagued by interferences with infrared devices in the classroom and infrared devices being used in adjacent rooms. Based on the instructors' experiences these drawbacks resulted in more disruptions in a classroom setting rather than providing substantive improvements.
4.2.2 Avoid the use of an off-site off-campus system administrator: Initially this sounds appealing. Based on the case study results, however, this causes more problems than efficiencies. Some vendors require registration at multiple levels (the university, the course, the instructor, the students and so on) EVERY semester. Requiring students to 'register' poses unique problems since (as an instructor) you would be requiring students to share their private information with a third-part vendor (e.g., name, e-mail ID, phone number, address, etc.). In addition, the vendor would have access to student scores from in-class questions and quizzes.
4.2.3 Do Not Underestimate Students' Resistance to Change: Recall from the survey results (Table 1), the non-adopting control section (section C) had unfavorable perceptions regarding the use of clickers in a classroom setting (2.70) and the ability of clickers to improve their overall performance in the class (2.85). These negative pre-hoc perceptions are in sharp contrast to the favorable impressions experienced by students that actually used the clickers (adopter sections A and B) in a classroom setting. These negative preconceptions may also present large obstacles towards reducing students' resistance to change. For example, students have long been accustomed to the inability to track attendance in large lecture courses. This fundamental student assumption is no longer valid and may take them by surprise. Some students schedule courses planning to attend class on exam days only, while other students may refuse to purchase the clickers or the textbook. Alternatively, some students become terribly upset for forgetting, losing or misplacing their clickers. Findings in the case study revealed the key is to establish clicker classroom policies early in the semester and to communicate the policies frequently. Additional effective approaches towards mitigating these issues is to offer students more opportunities to earn participation / attendance points than actual points required (the ratio used by the case study instructors has grown to nearly 2:1), or to provide in-class sign-in sheets for students forgetting their clickers.
4.2.4 Be cautious of new student integrity issues that emerge with the use of clickers: For example, one student may bring several of her friends' clickers to class and respond for each. Other students may swap clickers during class or simply share answers prior to responding. Instructors should establish policies addressing these issues in advance and communicate the policies to students. Two effective techniques discovered during the case study include periodic comparisons of student response counts (from CRS) to actual student head-count in the class-room, and the inclusion of on-the-fly questions (based on in-class discussions from the day) that strictly adhere to student response countdown timeframes. Thus, if students only have 15 seconds to respond to a newly prepared question, there is little time for clicker swapping or extended chatter between students. Other techniques could include mid-lecture RF frequency channel changes or mid-semester re-registrations of student clicker RFID numbers.
4.2.5 Minimize the clicker cost to students: The cost to students for clicker devices can vary widely depending on several factors. Does the textbook publisher offer bundled pricing with the text and clicker? Does the software vendor charge additional fees for students' reuse of the clickers on a per-semester or per-course basis? What is the bookstore's buyback/resale policy of clickers? Are the clickers IR or RF based? Does the university have a campus-wide agreement with a clicker software vendor? (If so, does the university assign them to students or do they expect students to purchase their own?) Several cost-benefit analyses (using various scenarios) were conducted during the case study. Rather then enumerate them all, here are some general rules of thumb:
* Although infrared clickers are significantly less expensive than RF, the performance problems associated with IR clickers is not worth the cost savings.
* Bundling clickers with textbooks offered the least expensive clicker cost to students. However, confirmation with the textbook publisher, the university bookstores and the CRS vendor is necessary.
* Additional per-semester or per-course fees are typically only incurred when an instructor makes use of the software vendor's central administration. As found in the case study, the performance problems associated with these off-campus central administration services are not worth the additional cost (even if they are offered for free).
* Determine if the university has established a recommended campus-wide clicker solution. Although these university-wide programs may cost students more up-front, they typically will cost students less over the longer term since the student can reuse their clicker in multiple classes.
4.3 Effective Practices
The following practices were found to be particularly effective during the CRS development, implementation and integration with the course curriculum.
4.3.1 Use the clickers as a real-life, real-time demonstration of an information system: The systems thinking perspective is traditionally one of the first concepts discussed in an introductory MIS course. The classroom response system (complete with a receiver dongle, the application software and the clickers) offers an excellent illustration of how various hardware and software components assembled together form an information system.
4.3.2 The greater the clicker usage levels, the greater the student perceived learning: Overall, the survey findings from the case study found that not only did student perceptions of clickers favorably influenced their performance in the course, the actual correlation between student attendance and performance did improve with greater levels of clicker usage (see Table 1). Furthermore, the results in Table 2 reveals that Section A students evaluated CRS more favorably than that of Section B students across all survey items. The most significant improvements areas from Section B to Section A occurred in core Active Learning and Motivational related survey items ("The use of clickers helps me prepare better for exams", "I learn more as a result of using the clickers" and "I remembered lecture material more as result of clickers" ). Robinson and Ritzko (2006) found similar learning and enjoyment improvements in their surveys of students enrolled in a smaller sized business course in a large university setting. In fact, 76% of their respondent students indicated they would be more likely to participate in a class opinion survey with the use of clickers, rather than raising their hand (Robinson and Ritzko, 2006).
4.3.3 Be prepared to explain why other answers are not correct: As instructors, we tend to focus on explaining (and defending) why the appropriate response to a question is correct. As a forewarning, also be prepared to fully explain why the other answers are not correct. Indeed, an important effective practice gained from the case study is to provide this explanation of incorrect responses in front of the entire class. Once the "ruling-out" explanations of incorrect responses was complete, did the instructors then reveal the correct answer and report how the class responded to the question. This provides essential insights and key exam preparation tactics for students and is consistently one of the greatest points of positive feedback regarding the system.
4.3.4 Wait a semester prior to heavy allocation of course points to clicker-based activities: It may take a semester before instructors (and students) become comfortable with using the system and the system becomes fully compatible with the technical environment. Thus, for the initial semester an instructor should consider allocating a smaller portion of participation / attendance points (3 to 5 percent of the overall course grade) associated with clicker-based activities. As the instructor gains confidence in the system, he or she should increase the frequency and broaden the scope of learning activities (in-class quizzes, group break-out sessions, etc.) that make use of CRS and their associated point value.
4.3.5 Align the clicker to curriculum integration with the effective learning concepts (active learning, providing feedback, attention span, motivation): A closer examination of how the adopter instructors from sections A and B of the case study utilized CRS provides additional insights into the favorable student survey results in Table 2. Both instructors, for example, used clickers for similar purposes (sample quiz questions, lecture comprehension checks, student surveys, opinion polls and attendance). Student feedback was substantially improved through CRS by enabling students to immediately view correct answers of lecture comprehension questions, permitting them to see how other students in the class responded, fully explaining why other answers were incorrect and promoting a learning environment by minimizing penalties associated with incorrect answers on clicker questions. Demetry (2006) found similar learner benefits in her study of using clickers as an effective formative assessment technique towards "closing the gap" between a learners' current state of understanding and Just-In Time teaching. Using similar techniques and with varying clicker usage (across three different instructional sessions), Hoffman and Goodwin (2006) found students benefited from substantially greater comprehension levels.
Student attention during lectures was significantly enhanced as well by using CRS to assist with transitioning between topics, altering the pace of lectures with periodic material comprehension questions, and surveying student opinions regarding a variety of controversial MIS topics. Active learning techniques were more fully employed with CRS through integrating the technology with in-class interactive discussions, guest speakers, student presentations, group breakout sessions, and with in-class textbook / lecture content questions and sample exam questions. Finally, student motivation was greatly rekindled by assigning points for student participation, student attendance, debating emerging IS topics, and striving to minimize student clicker costs while maximizing student value of effectively integrating CRS in the course curriculum.
4.3.6 Utilize in-class virtual break-out groups to stir motivation and competition: For example, most CRS products enable student responses to be categorized along a variety of demographics (males versus females, by major, by class, etc.) and to accumulate scores throughout a session. Thus, in-class questions pertaining to the privacy / ethics chapter could pin the different majors against one another (accountants versus finance versus management versus MIS). In-class questions pertaining to e-commerce topics could be used to examine differences between male versus female buyers. It is clear that in-class break-out groups can be used effectively to encourage diversity in the approach to teaching and to sustain student interest, with an enjoyable and enriched active learning environment.
4.3.7 Take student opinion polls: The IT industry is changing at an increasing rate. Countless controversial news stories emerge on a daily basis in matters concerning personal privacy, ethics, SPAM mail, viruses, new product launches, new website launches, search engines, IT vendors and so on. Use the system to take the temperature of student opinions on these controversial subjects and show the results to the class. The case study instructors found these simple polls to be excellent means of transitioning to different lecture topics, changing the pace of the lecture, and grabbing student attention at the start of class.
Figure 1--Embedding RF Clickers in Large Lecture--Case Study Findings Benefits --Improved student perceived performance in the course --Improved student class attendance and participation --Improved correlation between attendance and course grade --Improved student perceived exam preparation --Improved student perceived learning & interest in lectures --Improved depth, breadth and speed of feedback to students Lessons Learned --Avoid using infrared-based clickers --Avoid the use of an off-site, off-campus system administrator --Do not underestimate students' resistance to change --Be cautious of new student integrity issues that will emerge --Minimize the clicker cost to students Effective Practices for Course Integration --Use the clickers as a real-life, real-time demonstration of an information system --The greater the clicker usage levels, the greater the student perceived learning --Be prepared to explain why other answers are not correct --Wait a semester prior to heavy allocation of course points to clicker-based activities --Align clicker to curriculum integration with effective learning concepts (active learning, feedback, attention span, motivation) --Utilize clickers with in-class virtual break-out groups to stir motivation and competition --Take student opinion polls_
The intent of this study is to assist instructors with overcoming challenges inherent in large lecture courses via the use of clickers in the classroom. A case study was presented that provided a detailed account of the development, pilot testing, modification, implementation and integration with the curriculum of a large lecture course. The case study also highlighted the benefits realized, lessons learned, and effective teaching practices gained from the experience. Based on the case study findings, the most important system features and traits (from a technical standpoint) that should be sought in a clicker system include those that are RF-based, with on-site system administration, and provides a CRS application that is compatible and has a consistent look and feel with instructor's presentation software. The case study also found that the effective use of CRS technology can fundamentally change the traditional large-lecture classroom environment for both students and instructors. Instructors benefit from an improved ability to track attendance, better manage in-class group break-out sessions, and more accurately gauge lecture comprehension levels and adjust teaching styles or content accordingly on a real time basis. It provides instructors a means to identify students who are struggling prior to (rather than after) examinations. Students benefit from increased interest and participation levels during classes, improved comprehension of lecture material, better preparation for exams and greater perceived performance in the class. The study also found that the use of clickers improved students' perceived performance in the course and classes with clickers experienced greater attendance levels and higher correlations of student performance and attendance. Furthermore, the greater the volume of clicker usage, the more favorable student perceptions were in terms of active learning, motivation, and providing feedback. Key effective practices towards integrating CRS into the course curriculum and classroom include being prepared to explain why other answers are not correct, waiting a semester prior to heavy allocation of points to clicker-based activities, and being creative in the use of clickers (e.g. student opinion polls, in-class virtual groups, in-class demonstrations and many others). Although efforts were expended to minimize limitations of the study, some do remain. Namely, the study's scope is limited to a single university setting and a single course. Also, variations in between section survey results could be attributable to causes other than clicker technology, such as differences in teaching styles, exam content, class size and other contextual factors. Future research projects should consider improving on these limitations, as well as consider a closer examination of correlations between applied active learning procedures and the impact on student evaluations. For example, how are these "pay-off" correlations impacted as a result of clicker-enhanced active learning techniques? Our hope is that this case and reflective discussion will provide essential insights into the successful development and effective integration of CRS for current and future adopters.
The authors would like to acknowledge Dr. Gerry McKean, the Midwest United States Association for Information Systems, the Mid-American Journal of Business and several reviewers for their support and feedback of this manuscript.
An early draft of this manuscript appeared in the inaugural proceedings of the Midwest United States Association for Information Systems (MWAIS) 2006 conference in Grand Rapids, MI. on May 5-6, 2006.
Alavi, M., Yoo, Y., and Vogel, D. (1997), "Using information technology to add value to management education." Academy of Management Journal, Vol. 40, No. 6, pp. 1310-1333.
Ames, C. and Archer, J. (1988), "Achievement goals in the classroom: Students' learning strategies and motivation processes." Journal of Educational Psychology, Vol. 80, No.3, pp. 260-267.
Anderson, J.R. (1995), Cognitive Psychology and its Implications. 4th ed. Freedman, NY. Azevedo, R. and Bernard, R. M. (1995), "A meta-analysis of the effects of feedback in computer-based instruction." Journal of Educational Computing Research, Vol. 13, No. 2, pp. 111-127.
Bandura, A. (1986), Social foundations of thought and action: A Social cognitive Theory. Prentice Hall, Englewood Cliffs, NJ.
Bangert-Downs, R. L., Kulik, C. C., Kulik, J. A., and Morgan, M.T. (1991), "The instructional effect of feedback in test-like events." Review of Educational Research, Vol. 61, No. 2, pp. 213-238.
Beatty, I. (2004), "Transforming student learning with classroom communication systems." Educause Research Bulletin, Vol. 3, pp. 2-13. Available at http://www.educause.edu/ir/library/pdf/ERB0403.pdf
Bergtrom, G. (2006), "Clicker Sets as Learning Objects." Interdisciplinary Journal of Knowledge and Learning Objects, Vol. 2, pp. 105-110.
Bonk, C. J., and Cunningham, D. J. (1998), "Chapter 2: Searching for learner-centered, constructivist, and sociocultural components of collaborative educational learning tools." In C. J. Bonk and K. S. King (Eds.), Electronic Collaborators: Learner-Centered Technologies for Literacy, Apprenticeship, and Discourse, pp. 25-50. Erlbaum, Mahwah, NJ.
Bonwell, C.C., and Eison, J.A. (1991), "Active learning: Creating excitement in the classroom." ASHE-ERIC Higher Education Report No. 1. The George Washington University, School of Education and Human Development: Washington, DC.
Demetry, C. (2006), "Use of Formative Assessment to Probe Student Conceptions of the Lever Rule," The American Society of Engineering Education (ASEE) Annual Conference & Exposition: Excellence in Education, June 18-21.
Frase, L. T. (1971), "Effect of incentive variables and type of adjunct question upon text learning." Journal of Educational Psychology, Vol. 58, pp. 266-272.
Hoffman, C. and Goodwin, S. (2006), "A clicker for your thoughts: technology for active learning." New Library World, Vol. 107, No. 9, pp. 422-433.
Johnstone, A. H. and Percival, F. (1976), "Attention breaks in lectures." Education in chemistry, Vol. 13, pp. 49-50.
Kluger, A. N. and DeNisi, A. (1996), "The effects of feedback interventions on performance: A historical review, meta-analysis, and a preliminary feedback intervention theory." Psychological Bulletin, Vol. 119, pp. 254-284.
Kulik, J. A., and Kulik, C. C. (1988), "Timing of feedback and verbal learning." Review of Educational Research, Vol. 58, No. 1, pp. 79-97.
Leidner, D and Jarvenpaa, S (1993), "The information age confronts education: Case Studies on Electronic Classrooms." Information Systems Research, Vol. 4, pp. 24-54.
Meyers, C. and Jones, T.B. (1993), Promoting Active Learning: Strategies for the College Classroom. Jossey-Bass, San Francisco.
Robinson, S. and Ritzko, J. (2006), "Increasing Student Engagement Through Electronic Response Devices." Allied Academies International Conference, Proceedings of the Academy of Educational Leadership, April, pp. 7982.
Sorcinelli, M. D. (1991), "Research findings on the seven principles." New Directions for Teaching and Learning, Vol. 47, pp. 13-25.
Thalheimer, W. (2003), "The learning benefits of questions." Retrieved February 17, 2006, from http://www.work learning.com/ma/PP_WP003.asp
Weiner, B. (1990), "History of motivational research in education." Journal of Educational Research, Vol. 82, pp. 616-622.
Matthew L. Nelson
Roslin V. Hauck
Department of Accounting and Business Information Systems
Illinois State University
Normal, IL 61790-5520, USA
Matthew L. Nelson is an Assistant Professor at Illinois State University. He's held various roles in the IT industry, most recently as a Senior Financial Analyst at EDS, prior to completing his Ph.D. from the University of Illinois. Nelson instructs courses in e-Business Management, Business Information Systems Consulting and Introduction to Information Systems.
Roslin V. Hauck is an Assistant Professor of Business Information Systems at Illinois State University. Her research interests include technology adoption, organizational change, and the use of technology for knowledge management and sharing. Her teaching interests include Introduction to Information Systems and Systems Analysis and Design.
Table 1--Results Across All Sections Course: Introduction to Management Information Systems Spring Semester (Jan through May 2006) Section A Clicker Usage High Usage Volume of clicker "questions" during semester 270 n = Respondents (students) 175 Student Responses (Overall Perceptions) (a) Do you prefer classes that require clickers? 3.70 Clickers will improve my performance in the course? 3.79 Student Performance (Actual) Average Actual Class Attendance 0.86 Attendance / Performance Correlation 0.18 (e) Avg Student Course Grade (out of 100%) 82% Section B Clicker Usage Low Usage Volume of clicker "questions" during semester 50 n = Respondents (students) 65 Student Responses (Overall Perceptions) (a) Do you prefer classes that require clickers? 3.37 (b) Clickers will improve my performance in the course? 3.29 (c) Student Performance (Actual) Average Actual Class Attendance 0.75 Attendance / Performance Correlation 0.14 Avg Student Course Grade (out of 100%) 78% Section C Clicker Usage No Usage Volume of clicker "questions" during semester 0 n = Respondents (students) 63 Student Responses (Overall Perceptions) (a) Do you prefer classes that require clickers? 2.70 (d) Clickers will improve my performance in the course? 2.85 (d) Student Performance (Actual) Average Actual Class Attendance 0.68 Attendance / Performance Correlation 0.12 Avg Student Course Grade (out of 100%) 79% (a) Measured on a E-point Likert Scale from Strongly Disagree (1) to Strongly Agree (5} (b) Significant difference in means between Section A and B at the p < .1 level (c) Significant difference in means between Section A and B at the p < .001 level (d) Significant difference in means between Section B and C at the p < .01 level (e) Significant correlation at the p < .01 level Table 2--Survey Item Results from Users Only (Sections A & B) Sec A & B Clicker Usage Combined Volme of clicker "questions" during semester 320 n = Respondents 240 Mean Scores ACTIVE LEARNING [alpha = .78} c6 The use of clickers improved my performance in the course 3.66 c1 I learn more as a result of using the clickers 3.76 c13 I remembered lecture material more as result of clickers 3.91 c12 I attend more classes as a result of clickers 3.99 MOTIVATION [alpha = .92) c3 The use of clickers is worth the extra effort 4.04 c7 I would like to see clickers used in other courses 3.77 c11 I recommend classes that use clickers to other students 3.84 C12 I prefer classes that use clicker technology 3.61 c16 I The value this system adds to course is worth the extra cost. 3.38 FEEDBACK [alpha = .62) c18 I like seeing how other students answered questions 3.96 c15 The use of clickers helps me prepare better for exams 3.78 c14 I prefer the use of clickers over the use of Opscan sheets 3.80 c17 The use of clickers is more accurate the Opscan sheets 3.26 ATTENTION SPAN [alpha = .61) c3 The use of clickers makes the class more interesting 3.94 c10 I participate in classes more as result of clickers 4.23 c4 The clicker technology is easy to use 4.61 Section A Section B Clicker Usage High Usage Low Usage Volme of clicker "questions" during semester 270 50 n = Respondents 175 65 Mean Scores ACTIVE LEARNING [alpha = .78} * c6 The use of clickers improved my performance in the course 3.79 3.29 * c1 I learn more as a result of using the clickers 3.94 3.26 * c13 I remembered lecture material more as result of clickers 4.0E 3.51 * c12 I attend more classes as a result of clickers 4.07 3.77 *** MOTIVATION [alpha = .92) c3 The use of clickers is worth the extra effort 4.1E 3.73 * c7 I would like to see clickers used in other courses 3.87 3.51 ** c11 I recommend classes that use clickers to other students 3.99 3.45 * C12 I prefer classes that use clicker technology 3.70 3.37 *** c16 I The value this system adds to course is worth the extra cost. 3.51 3.03 ** FEEDBACK [alpha = .62) c18 I like seeing how other students answered questions 3.97 3.95 ns c15 The use of clickers helps me prepare better for exams 3.99 3.23 * c14 I prefer the use of clickers over the use of Opscan sheets 3.93 3.45 ** c17 The use of clickers is more accurate the Opscan sheets 3.34 3.03 *** ATTENTION SPAN [alpha = .61) c3 The use of clickers makes the class more interesting 4.00 3.77 ns c10 I participate in classes more as result of clickers 4.28 4.09 ns c4 The clicker technology is easy to use 4.62 4.E0 ns Nates:--Items measured an a E-paint Likert Scale from Strongly Disagree (1} through Strongly Agree (5) * Significant Difference in Means p<.001 ** Significant Difference in Means p<.010 *** Significant Difference in Means p<.100…
Questia, a part of Gale, Cengage Learning. www.questia.com
Publication information: Article title: Clicking to Learn: A Case Study of Embedding Radio-Frequency Based Clickers in an Introductory Management Information Systems Course. Contributors: Nelson, Matthew L. - Author, Hauck, Roslin V. - Author. Journal title: Journal of Information Systems Education. Volume: 19. Issue: 1 Publication date: Spring 2008. Page number: 55+. © EDSIG Fall 2008. COPYRIGHT 2008 Gale Group.
This material is protected by copyright and, with the exception of fair use, may not be further copied, distributed or transmitted in any form or by any means.