How can you tell when a marketplace is exploding? Maybe when the benchmark evaluation of that marketplace can no longer contain it. Just ask Gartner, which decided in the wake of its 2009 Magic Quadrant for Social Software report to kill off its Magic Quadrant for Social Software report. Like a Hydra, though, new reports sprouted in its place, and the analyst firm took on the familiar chore of assessing (and, notably, excluding) vendors in not one but three segments of social software: Social CRM (SCRM), Social Software for the Workplace, and Externally Facing Social Software (EFSS). Divide and conquer? More like be fruitful and multiply: Most vendor evaluations and rankings leave some incumbents as well as rising stars feeling sideswiped, overlooked, or even misjudged. But each of these reports, perhaps due to the newness of the technologies involved, sparked three times the controversy of its single predecessor.
Gartner Research Director Adam Sarner says the response was expected--even welcome. "We have an opinion of where the market is now and it's backed up by research," Sarner insists. "What the vendors say is important to us and we will continue to listen as the market evolves. For now, we shouldn't ignore any input that we get."
One of the harshest criticisms--from vendors as well as industry watchdogs--was about the lack of attention paid to social media monitoring. Sarner responds that Gartner expects to soon see the micro-segment--comprising Radian6, Attensity Group, and others--swallowed up by the larger social software market.
Quibbling over omissions is one thing, but even the placement of included vendors has raised eyebrows. In two of the new reports--SCRM and EFSS--Gartner named only Jive Software and Lithium Technologies as Leaders. With only a few players in each report named Visionaries, most companies ended up crammed together as Niche Players, which Gartner describes as vendors that are narrow in scope yet "provide useful, focused technology." Critics argue that Niche Players are perhaps too differentiated to be evaluated in the same report.
Esteban Kolsky, founder of CRM consultancy ThinkJar (and a former Gartner analyst), says a MarketScope--a less-rigorous format Gartner applies to less-mature markets--would have been more useful here. "Then they wouldn't put 19 vendors [on a Magic Quadrant] and attempt to score them all equally," he says. "I've always questioned the validity of Magic Quadrants as a determinant in who is a market leader and who is not."
There's also, Kolsky laments, a mindset in the technology field that a marketplace can't possibly be considered "well established" until Gartner does a Quadrant. And yet Gartner hadn't released one specifically on social CRM until now, despite exploding market demand. Kolsky says the Gartner CRM Summit in 2009 hardly addressed the concept of social CRM, and yet this spring's Gartner Customer 360 event--just a few months later--devoted several keynotes to social CRM. …