This paper describes a methodology based on 40 criterions as well as a mixture of MCDM methods to evaluate different technologies applied in modern training systems. The criterions are grouped in a three-dimensional (3D) model in accordance with their use and application in training processes. The proposed 3D model includes the Management (M), the Technological (T) and the Instructional (I) dimensions. Applying this methodology we can evaluate different training and learning technologies. Furthermore a technology roadmap is outlined using the obtained results in the evaluation. This roadmap will provide CFE (Comisión Federal de Electricidad which is the National Electricity Utility in Mexico) with elements to decide, enhance and upgrade its current training information systems or acquire new ones as well as showing what courses are worth implementing with e-learning technologies and when these courses should be programmed.
Key words: Roadmap, e-learning, learning objects, evaluation, MCDM, methodology
(ProQuest-CSA LLC: ... denotes formulae omitted.)
The 3D model and the proposed methodology in this paper, not only are helpful to evaluate the applicability of each learning tool from a global point of view, but they are also useful to establish the utilization of each learning tool in every dimension (M, T, I), in every plane (MT, MI, TI) and in a 3-dimensional space (MTI). This provides different views of the same tool, which allows evaluating each tool from different perspectives. These perspectives help to determine whether or not a tool fulfills the requirements from an M-T-I point of view.
The management dimension evaluates aspects such as: Course Management, Student Tracking, Curriculum Management, etc. The instructional dimension includes Instructional Design, Didactic Planning, Content Production, Instructor Manual, Student Manual, and so on. Finally, the technological dimension involves the current software and hardware tools available as candidates for acquisition and use in the training processes of a company.
These 3 conceptual dimensions outline a 3D space and three planes (see Figure 1). All of these represent different relationships among them. The elements of the management-technological plane or quadrant relate how technology is being applied in course administration and how much the cost of hardware and software is. The aspects of the management-instructional quadrant relate how we can administrate the content of courses and keep track of student's progress and performance of his/her training activities. The issues in the technological-instructional plane relate how software and hardware technology is used to develop content for courses. And finally the management technological-instructional space indicates how the technology is being used to manage the learning process.
Commercially available tools integrate different modules providing a complete learning tool. However, this integration increases the cost and complexity of each tool. In this paper we present evaluation results for some Learning Management Systems (LMSs), Learning Content Management Systems (LCMSs) and Content Management Systems (CMSs). The CMSs are complementary tools to obtain a more complete learning tool.
The proposed methodology was used to evaluate three commercial platforms (Blackboard, IBM Lotus and PeopleSoft) and two in-house developed tools (SIC, and UTEC). The SIC and UTEC tools are systems used by CFE to manage its Human Capital to align workforce with business objectives.
The rest of the paper is organized as follows: section 2 summarizes a review of evaluation methodologies; section 3 includes some e-learning concepts related to the tools evaluated in this paper; section 4 describes the evaluation methodology used; section 5 illustrates how the methodology is applied to evaluate e-learning tools, also shows the evaluation outcomes and includes an e-learning roadmap slide, which is based on the evaluation outcomes; section 6 includes some conclusions and finally section 7 provides a list of references. …