factor they considered more important. If they considered both factors in a pair equally important, they were to circle both alternatives. To calculate the degree of agreement between the subjects (inter-rater reliability), the method of the Kendall coefficient of agreement u for paired comparisons was used ( Siegel and Castellan 1988).
The subjects were also asked to rate each factor on a scale from not at all important (1) to extremely important (5).
Fifteen subjects made the paired comparisons. The single comparisons were compiled into a preference matrix showing the distribution and the sums of preferences (Table 1). Based on the matrix, Kendall's u was calculated. The results (u = 0.097, X2 = 35.27, df = 15) show a statistically significant agreement between the comparisons by the subjects (p < 0.01). The interpersonal preferences for the different factors can be concluded to be consistent.
Table 1. Preference matrix based on the answers by the subjects (N = 15) (number of subjects preferring the row factor to the column one)
Further, the votes for each factor from Table 1 were converted into factor weights by calculating the vote proportion for each (Table 2). The proportions of all the six factors constitute a total of 1.0. According to Table 2, the most important factor was training, while planning ranked second. The results of rating also (Table 2) pointed out the importance of training. System developers and planning came next.