I must again repeat, what the assailants of utilitarianism seldom have the justice to acknowledge, that the happiness which forms the utilitarian standard of what is right in conduct, is not the agent’s own happiness, but that of all concerned. As between his own happiness and that of others, utilitarianism requires him to be as strictly impartial as a disinterested and benevolent spectator.
(2.18; cf. 5.36, including n.)
It is ironic that Mill’s defence of utilitarianism against the charge that it is egoistic encapsulates particularly well what many modern writers think is mistaken about utilitarianism. According to utilitarianism, all that matters is welfare and its maximization: whose welfare is irrelevant. That is to say, utilitarianism is concerned primarily not with the distribution of welfare, but with its aggregation.