A Tutorial on MM Algorithms

By Hunter, David R.; Lange, Kenneth | The American Statistician, February 2004 | Go to article overview

A Tutorial on MM Algorithms


Hunter, David R., Lange, Kenneth, The American Statistician


1. INTRODUCTION

Maximum likelihood and least squares are the dominant forms of estimation in frequentist statistics. Toy optimization problems designed for classroom presentation can be solved analytically, but most practical maximum likelihood and least squares estimation problems must be solved numerically. This article discusses an optimization method that typically relies on convexity arguments and is a generalization of the well-known EM algorithm method (Dempster, Laird, and Rubin 1977; McLachlan and Krishnan 1997). We call any algorithm based on this iterative method an MM algorithm.

To our knowledge, the general principle behind MM algorithms was first enunciated by the numerical analysts Ortega and Rheinboldt (1970) in the context of line search methods. De Leeuw and Heiser (1977) presented an MM algorithm for multidimensional scaling contemporary with the classic Dempster et al. (1977) article on EM algorithms. Although the work of de Leeuw and Heiser did not spark the same explosion of interest from the statistical community set off by the Dempster et al. (1977) article, steady development of MM algorithms has continued. The MM principle reappears, among other places, in robust regression (Huber 1981), in correspondence analysis (Heiser 1987), in the quadratic lower bound principle of Bohning and Lindsay (1988), in the psychometrics literature on least squares (Bijleveld and de Leeuw 1991; Kiers and Ten Berge 1992), and in medical imaging (De Pierro 1995; Lange and Fessler 1995). The recent survey articles of de Leeuw (1994), Heiser (1995), Becket, Yang, and Lange (1997), and Lange, Hunter, and Yang (2000) deal with the general principle, but it is not until the rejoinder of Hunter and Lange (2000a) that the acronym MM first appears. This acronym pays homage to the earlier names "majorization" and "iterative majorization" of the MM principle, emphasizes its crucial link to the better-known EM principle, and diminishes the possibility of confusion with the distinct subject in mathematics known as majorization (Marshall and Olkin 1979). Recent work has demonstrated the utility of MM algorithms in a broad range of statistical contexts, including quantile regression (Hunter and Lange 2000b), survival analysis (Hunter and Lange 2002), paired and multiple comparisons (Hunter 2004), variable selection (Hunter and Li 2002), and DNA sequence analysis (Sabatti and Lange 2002).

One of the virtues of the MM acronym is that it does double duty. In minimization problems, the first M of MM stands for majorize and the second M for minimize. In maximization problems, the first M stands for minorize and the second M for maximize. (We define the terms "majorize" and "minorize" in Section 2.) A successful MM algorithm substitutes a simple optimization problem for a difficult optimization problem. Simplicity can be attained by (a) avoiding large matrix inversions, (b) linearizing an optimization problem, (c) separating the parameters of an optimization problem, (d) dealing with equality and inequality constraints gracefully, or (e) turning a nondifferentiable problem into a smooth problem. Iteration is the price we pay for simplifying the original problem.

In our view, MM algorithms are easier to understand and sometimes easier to apply than EM algorithms. Although we have no intention of detracting from EM algorithms, their dominance over MM algorithms is a historical accident. An EM algorithm operates by identifying a theoretical complete data space. In the E-step of the algorithm, the conditional expectation of the complete data log-likelihood is calculated with respect to the observed data. The surrogate function created by the E-step is, up to a constant, a minorizing function. In the M-step, this minorizing function is maximized with respect to the parameters of the underlying model; thus, every EM algorithm is an example of an MM algorithm. Construction of an EM algorithm sometimes demands creativity in identifying the complete data and technical skill in calculating an often complicated conditional expectation and then maximizing it analytically. …

The rest of this article is only available to active members of Questia

Already a member? Log in now.

Notes for this article

Add a new note
If you are trying to select text to create highlights or citations, remember that you must now click or tap on the first word, and then click or tap on the last word.
One moment ...
Default project is now your active project.
Project items

Items saved from this article

This article has been saved
Highlights (0)
Some of your highlights are legacy items.

Highlights saved before July 30, 2012 will not be displayed on their respective source pages.

You can easily re-create the highlights by opening the book page or article, selecting the text, and clicking “Highlight.”

Citations (0)
Some of your citations are legacy items.

Any citation created before July 30, 2012 will labeled as a “Cited page.” New citations will be saved as cited passages, pages or articles.

We also added the ability to view new citations from your projects or the book or article where you created them.

Notes (0)
Bookmarks (0)

You have no saved items from this article

Project items include:
  • Saved book/article
  • Highlights
  • Quotes/citations
  • Notes
  • Bookmarks
Notes
Cite this article

Cited article

Style
Citations are available only to our active members.
Buy instant access to cite pages or passages in MLA, APA and Chicago citation styles.

(Einhorn, 1992, p. 25)

(Einhorn 25)

1. Lois J. Einhorn, Abraham Lincoln, the Orator: Penetrating the Lincoln Legend (Westport, CT: Greenwood Press, 1992), 25, http://www.questia.com/read/27419298.

Cited article

A Tutorial on MM Algorithms
Settings

Settings

Typeface
Text size Smaller Larger Reset View mode
Search within

Search within this article

Look up

Look up a word

  • Dictionary
  • Thesaurus
Please submit a word or phrase above.
Print this page

Print this page

Why can't I print more than one page at a time?

Help
Full screen

matching results for page

    Questia reader help

    How to highlight and cite specific passages

    1. Click or tap the first word you want to select.
    2. Click or tap the last word you want to select, and you’ll see everything in between get selected.
    3. You’ll then get a menu of options like creating a highlight or a citation from that passage of text.

    OK, got it!

    Cited passage

    Style
    Citations are available only to our active members.
    Buy instant access to cite pages or passages in MLA, APA and Chicago citation styles.

    "Portraying himself as an honest, ordinary person helped Lincoln identify with his audiences." (Einhorn, 1992, p. 25).

    "Portraying himself as an honest, ordinary person helped Lincoln identify with his audiences." (Einhorn 25)

    "Portraying himself as an honest, ordinary person helped Lincoln identify with his audiences."1

    1. Lois J. Einhorn, Abraham Lincoln, the Orator: Penetrating the Lincoln Legend (Westport, CT: Greenwood Press, 1992), 25, http://www.questia.com/read/27419298.

    Cited passage

    Thanks for trying Questia!

    Please continue trying out our research tools, but please note, full functionality is available only to our active members.

    Your work will be lost once you leave this Web page.

    Buy instant access to save your work.

    Already a member? Log in now.

    Author Advanced search

    Oops!

    An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.