John Aubrey (1626-97) was educated at Trinity College, Oxford, and even from his time as an undergraduate developed his historical interests. The Brief Lives were never published in his own time, when his reputation was strictly tied to his achievement as an antiquarian. Outside the Life, other mentions of Spenser occur in his accounts of Sidney (repeating first the story about the Despair Canto), of the Countess of Pembroke (to the effect that he was not a stranger at Wilton), and of Michael Drayton.
From Brief Lives, in the edition of O.L. Dick (1949), pp. 282-3:
Mr Beeston sayes, he was a little man, wore short haire, little band and little cuffs.
Mr Edmund Spencer was of Pembroke-hall in Cambridge; he misst the Fellowship there, which Bishop Andrewes gott. He was an acquaintance and frequenter of Sir Erasmus Dreyden: His Mistris Rosalind was a kinswoman of Sir Erasmus Lady s. The chamber there at Sir Erasmus' is still called Mr Spencers chamber. Lately, at the college takeing-downe the Wainscot of his chamber, they found an abundance of Cards, with stanzas of the Faerie Queen written on them.
Mr Samuel Woodford (the Poet who paraphras'd the Psalmes) lives in Hampshire neer Alton, and he told me that Mr Spenser lived sometime in these parts, in this delicate sweet ayre: where he enjoyed his Muse: and writt good part of his Verses. He had lived some time in Ireland, and made a description of it, which is printed.
I have said before that Sir Philip Sidney, and Sir Walter Ralegh were his acquaintance. Sir John Denham told me, that ABp. Usher, Lord Primate of Armagh, was acquainted with him; by this token: when Sir William Davenant's Gondibert came forth, Sir John askt the Lord Primate if he had seen it. Said the Primate, Out upon him, with his vaunting Preface, he speakes against my old friend Edmund Spenser.