How can one evaluate the performance of federal appellate judges? This question implicitly arises every time a federal appellate judge is nominated to the United States Supreme Court. And because the federal appellate bench is the most common source of Supreme Court nominees in recent decades, (1) this question is relevant to most modern Supreme Court nominations. But the question of judicial performance is at least as important outside the context of Supreme Court appointments, as the courts of appeals are the final arbiters of most disputes in the federal courts. Thus, the outcome of virtually every litigated matter in the federal system hinges on the quality of federal appellate decision-making, and therefore the performance of these judges implicates fundamental questions about the rule of law.
However, the importance of evaluating the performance of federal judges has not motivated systematic assessment of individual judges' work product in legal scholarship. Indeed, aside from anecdotal information, little is known about the performance of individual federal appellate judges. of course there is no dearth of scholarly critique of federal courts' products--the opinions in individual cases--but these critiques are not systematically organized into evaluation of the producers of the opinions--the judges themselves. Thus, in spite of the fact that the performance of individual judges has important implications for the functioning of the judicial system and rule of law, scholars still do not have a good idea of which judges are performing well and which judges are performing poorly. Few academic studies have even attempted to evaluate federal judges using quantitative data, and, when they have, they have generally received harsh criticism from scholarly commentators. (2)
The recent nomination of then-Judge Sonia Sotomayor to the Supreme Court illustrates the de facto alternative to systematic approaches to judicial quality. The Sotomayor nomination, like nominations of federal appellate judges in the past, tended to focus on detailed scrutiny of a small number of high-profile opinions, distracting from the broader, systematic examination of the nominee's body of work as a whole. (3) In the absence of reliable information about judicial performance, center stage in the debate is yielded to anecdotal accounts of anonymous sources, (4) isolated remarks from the judge's public appearances, and short passages in opinions culled from the tens of thousands of pages the nominee has written. Although we have the benefit of a more thorough evaluation from the American Bar Association, its approach has been called biased (5) and may be no more objective than the confirmation hearings. The result is that the evaluation of judicial performance is biased, subjective, and based on a narrow slice of information rather than on the judge's record as a whole.
The frustration with the prevailing approaches to assessing judicial quality, both in the context of Supreme Court appointments and otherwise, has led scholars and legal commentators to develop quantitative techniques to measure judicial performance. (6) The most prominent approaches in recent years use large databases of citations to evaluate the "influence," "prestige," or "quality" of judges. One of the first such papers, by Professors Landes, Lessig, and Solimine (hereinafter "Landes et al."), used citation counts to opinions to measure "judicial influence" in the federal courts of appeals. (7) More recently, Professors Choi and Gulati have expanded on the Landes et al. study, using citation counts to measure "productivity," "quality," and "independence" on the federal courts of appeals. (8) In contrast to the typical evaluation of judges' opinions that legal scholars perform in law reviews, the citation literature abstracts away from the details of the cases to systematically evaluate the whole body of the judges' work product. …