Evaluating the Web
Powers, Mike, Human Ecology Forum
Corporations and organizations are making huge investments to create World Wide Web sites. But most are making the leap into cyberspace with little or no knowledge of what impact their sites will have. Are their investments worth it? Human Ecology professor William Trochim is trying to find out.
Web sites are the hula hoop of the 90s for today s businesses and organizations. Almost everybody has one, and those who don't desperately want one.
But are they worth it?
"A lot of them may be wasting their time and money," says policy analysis and management professor William Trochim. "I don't think we have a lot of evidence yet about how Web sites affect what people do and what the return might be for investing in the technology. People do it because they feel they have to do it."
Trochim is an expert in program evaluation and social science research methodology. He uses the tools of his trade to evaluate the effectiveness of research methods, policy, and other projects that involve complex planning.
"The idea is to identify the outcomes and effects people are interested in," he says. "We also want to identify outcomes they may not have foreseen."
Trochim has turned his critical eye to cyberspace. He's in the early stages of figuring out how he can apply the evaluation tools he's honed over the years to the analysis of Web sites.
He has plenty of candidates to look at. It's virtually impossible today to find a corporation that hasn't created a Web site. Many are quite extensive, with dozens and even hundreds of pages. Some companies have hired large staffs to design and maintain their sites, many of which are updated daily. Others pay freelancers an average of $800 a page to do the work. Either way, these companies are making sizable investments to have a presence in cyberspace.
Once they have their sites up and running, the first question is whether people are visiting them. That answer is easily obtained. Software programs that count "hits," the number of times a site has a visitor, are common features of many Web sites these days. They appear much like the odometer on a car, and a visitor to a site can tell exactly how many other people have previously visited the site.
But Trochim says there's a much more important question to be answered. What, if any, are the effects of a site on the behavior of its visitors? That answer is much harder to determine.
"There's a big difference between evaluating how a Web site is used and evaluating the effects it has on other factors that aren't directly measurable over the Web," he says. "For example, in a classroom context we can evaluate how students use the Web. We can see when they use it, see what pages they're hitting, and count hit rates. But I don't just want to know what sites the students are visiting. I want to know how their use of the Web relates to how they perform, how it affects their grades, and whether or not they attend class."
That sort of information can be even more important to businesses, he adds. They want to know how use of the Web affects their employees' performance on the job. In their case, dollars, not grades, are on the line.
"So the big question is not what people are doing on the Web, it's how do you link that data to external measurements. It's a big challenge, but that's where the payoffs are going to be. So far we have very little experience doing this."
As a first step in Web site evaluation, Trochim is engaging in a bit of critical self-analysis. By his own admission a closet computer hacker, he jumped on the Web bandwagon early on, creating an extensive site, "Bill Trochim's Center for Social Research Methods." It describes his own research and includes many of his papers, links users to other resources, and provides an electronic forum where people can exchange ideas and even organize projects.
Part of the site is a two-year-old Course Center, which houses teaching sites for the three courses he conducts in research design and program evaluation and planning. …