Underpinnings of System Evaluation
David W. Abbott
Mark A. Wise
University of Central Florida
John A. Wise
Embry-Riddle Aeronautical University
Rapid advances in software and hardware have provided the capability to develop very complex systems that have highly interrelated components. Although this has permitted significant increases in system efficiency and has allowed the development and operation of systems that were previously impossible (e.g., negative stability aircraft), it has also brought the danger of system-induced catastrophes. Perrow ( 1984) argued that complex systems that are highly coupled (i.e., have highly interdependent components) are unstable and have a disposition toward massive failure. This potential instability makes human-factors-based evaluation more important than it has been in the past, whereas the component coupling makes the traditional modular evaluation methods obsolete.
Systems that are highly coupled can create new types of failures. The coupling of components that were previously independent can result in unpredicted failures. As systems become more coupled, interdisciplinary issues will become more critical. For example, it is possible that new problems could reside in the human--machine interface where disciplines meet and interact. It is in these intellectual intersections that new compromises and cross-discipline trade-offs will be made. And it will be in these areas that new and unanticipated human-factors-based failures may emerge.
As systems grow in complexity and intradependence the cost of performing adequate testing is rapidly approaching a critical level. The cost of certification in aviation has been a significant cost driver. The popular aviation press is continually carrying articles on an aviation part (e.g., an alternator) that is exactly the same as an automobile part (i.e., comes off exactly the same assembly line) but that costs two to three times as much because of the aviation certification costs. Human-factors-based verification, validation, and certification methods must thus not only be effective, they must also be cost-effective.
"Technically adequate" human factors testing may not even be sufficient or even relevant to a system becoming safely operational. The political and emotional issues