# A Tutorial on MM Algorithms

By Hunter, David R.; Lange, Kenneth | The American Statistician, February 2004 | Go to article overview

# A Tutorial on MM Algorithms

Hunter, David R., Lange, Kenneth, The American Statistician

1. INTRODUCTION

Maximum likelihood and least squares are the dominant forms of estimation in frequentist statistics. Toy optimization problems designed for classroom presentation can be solved analytically, but most practical maximum likelihood and least squares estimation problems must be solved numerically. This article discusses an optimization method that typically relies on convexity arguments and is a generalization of the well-known EM algorithm method (Dempster, Laird, and Rubin 1977; McLachlan and Krishnan 1997). We call any algorithm based on this iterative method an MM algorithm.

To our knowledge, the general principle behind MM algorithms was first enunciated by the numerical analysts Ortega and Rheinboldt (1970) in the context of line search methods. De Leeuw and Heiser (1977) presented an MM algorithm for multidimensional scaling contemporary with the classic Dempster et al. (1977) article on EM algorithms. Although the work of de Leeuw and Heiser did not spark the same explosion of interest from the statistical community set off by the Dempster et al. (1977) article, steady development of MM algorithms has continued. The MM principle reappears, among other places, in robust regression (Huber 1981), in correspondence analysis (Heiser 1987), in the quadratic lower bound principle of Bohning and Lindsay (1988), in the psychometrics literature on least squares (Bijleveld and de Leeuw 1991; Kiers and Ten Berge 1992), and in medical imaging (De Pierro 1995; Lange and Fessler 1995). The recent survey articles of de Leeuw (1994), Heiser (1995), Becket, Yang, and Lange (1997), and Lange, Hunter, and Yang (2000) deal with the general principle, but it is not until the rejoinder of Hunter and Lange (2000a) that the acronym MM first appears. This acronym pays homage to the earlier names "majorization" and "iterative majorization" of the MM principle, emphasizes its crucial link to the better-known EM principle, and diminishes the possibility of confusion with the distinct subject in mathematics known as majorization (Marshall and Olkin 1979). Recent work has demonstrated the utility of MM algorithms in a broad range of statistical contexts, including quantile regression (Hunter and Lange 2000b), survival analysis (Hunter and Lange 2002), paired and multiple comparisons (Hunter 2004), variable selection (Hunter and Li 2002), and DNA sequence analysis (Sabatti and Lange 2002).

One of the virtues of the MM acronym is that it does double duty. In minimization problems, the first M of MM stands for majorize and the second M for minimize. In maximization problems, the first M stands for minorize and the second M for maximize. (We define the terms "majorize" and "minorize" in Section 2.) A successful MM algorithm substitutes a simple optimization problem for a difficult optimization problem. Simplicity can be attained by (a) avoiding large matrix inversions, (b) linearizing an optimization problem, (c) separating the parameters of an optimization problem, (d) dealing with equality and inequality constraints gracefully, or (e) turning a nondifferentiable problem into a smooth problem. Iteration is the price we pay for simplifying the original problem.

In our view, MM algorithms are easier to understand and sometimes easier to apply than EM algorithms. Although we have no intention of detracting from EM algorithms, their dominance over MM algorithms is a historical accident. An EM algorithm operates by identifying a theoretical complete data space. In the E-step of the algorithm, the conditional expectation of the complete data log-likelihood is calculated with respect to the observed data. The surrogate function created by the E-step is, up to a constant, a minorizing function. In the M-step, this minorizing function is maximized with respect to the parameters of the underlying model; thus, every EM algorithm is an example of an MM algorithm. Construction of an EM algorithm sometimes demands creativity in identifying the complete data and technical skill in calculating an often complicated conditional expectation and then maximizing it analytically. …

If you are trying to select text to create highlights or citations, remember that you must now click or tap on the first word, and then click or tap on the last word.
One moment ...
Default project is now your active project.
Project items

Highlights (0)
Some of your highlights are legacy items.
Citations (0)
Some of your citations are legacy items.
Notes (0)
Bookmarks (0)

Project items include:
• Saved book/article
• Highlights
• Quotes/citations
• Notes
• Bookmarks
Notes

#### Cited article

Style
Citations are available only to our active members.

(Einhorn, 1992, p. 25)

(Einhorn 25)

1. Lois J. Einhorn, Abraham Lincoln, the Orator: Penetrating the Lincoln Legend (Westport, CT: Greenwood Press, 1992), 25, http://www.questia.com/read/27419298.

#### Cited article

A Tutorial on MM Algorithms
Settings

#### Settings

Typeface
Text size Reset View mode
Search within

Look up

#### Look up a word

• Dictionary
• Thesaurus
Please submit a word or phrase above.

Why can't I print more than one page at a time?

Help
Full screen

### How to highlight and cite specific passages

1. Click or tap the first word you want to select.
2. Click or tap the last word you want to select, and you’ll see everything in between get selected.
3. You’ll then get a menu of options like creating a highlight or a citation from that passage of text.

## Cited passage

Style
Citations are available only to our active members.

"Portraying himself as an honest, ordinary person helped Lincoln identify with his audiences." (Einhorn, 1992, p. 25).

"Portraying himself as an honest, ordinary person helped Lincoln identify with his audiences." (Einhorn 25)

"Portraying himself as an honest, ordinary person helped Lincoln identify with his audiences."1

1. Lois J. Einhorn, Abraham Lincoln, the Orator: Penetrating the Lincoln Legend (Westport, CT: Greenwood Press, 1992), 25, http://www.questia.com/read/27419298.

## Thanks for trying Questia!

Please continue trying out our research tools, but please note, full functionality is available only to our active members.

Your work will be lost once you leave this Web page.