JUDICIAL PERFORMANCE EVALUATION: Steps to Improve Survey Process and Measurement
Elek, Jennifer K., Rottman, David B., Cutler, Brian L., Judicature
In the wake of charges that judicial performance evaluation surveys are biased against women and minority judges, states must take action to ensure that programs are fair and equitable to all.
State judicial performance evaluation (JPE) programs promise to help courts achieve a variety of central goals (e.g., more informed judicial selection, retention, and/or assignment decisions; improvements in judicial quality; greater transparency). However, recent criticisms leveled against these programs and supported by preliminary empirical evidence portray JPE surveys based on the popular ABA model1 as systematically biased against minority and women judges.2 Such claims invariably heighten the methodological scrutiny applied to all survey-based JPE programs that, in turn, will likely reveal a number of other shortcomings in existing JPE surveys. Most state JPE surveys do not reflect recent advances in the scientific understanding of survey design related to performance evaluation. States must remedy weaknesses in their JPE surveys if they wish to preserve the credibility of JPE programs in the public's eye and within the court community. To provide states with some guidance in this effort, we review several fundamental shortcomings common to state and model JPE surveys in the U.S. and offer some concrete steps for improvement in key areas.
An Overview of Judicial Performance Evaluation in the States
The first official state-sponsored judicial performance evaluation program began in Alaska in 1976 as part of an effort to address concerns that the voting public lacked sufficient information to make educated decisions about judges in retention elections.3 Many other states followed suit: A 2004 national survey identified 21 states and territories with official JPE programs and 1 state with a pilot program.4 The Institute for the Advancement of the American Legal System (IAALS) website recognizes 18 states that presently have active JPE programs.5 The specific purposes of these programs vary by state: Results may be disseminated to judges to facilitate self-improvement, to the public to facilitate more informed voting decisions, and to judicial administration to facilitate more effective retention decisions and inform other administrative decision-making processes. Proponents of JPEs point to the potential of these programs to improve the quality of justice, to reinforce judicial independence, and to foster greater public trust and confidence in the judiciary, among other benefits.6
Whether JPE programs reach their full potential depends on a variety of factors.7 The foundation of a strong JPE program rests perhaps most heavily on the quality of metrics used. Any state seeking to create or improve upon an existing JPE program faces a number of critical decisions about the substantive content and execution of the performance evaluation itself. Early in the development process, for example, those responsible for a state JPE program must choose which of the available performance measures and methods to employ. These may include but are not limited to caseload and workload statistics, courtroom observations, personal interviews, substantive reviews [e.g., of the judge's decisions, opinions, or orders; of past disciplinary actions against the judge), feedback from the public, and/or surveys of court users and employees.8
Of the available methods, nearly every state incorporates some form of evaluation survey.9 Undoubtedly, the popularity of the survey method is due in part to the fact that surveys can be used to gather a standard set of information from a large group of individuals in a relatively short amount of time. A well-designed survey is a powerful and efficient data-collection tool. However, poorly designed surveys can produce misleading or useless information.
Despite the many useful conceptual guides that have been proffered to advise states on JPE program development and to outline the many indicators of good judicial performance,10 little detailed technical guidance is publicly available to states on how to craft a good JPE survey instrument. …