I'd pick Frazier for that one

Holmes is an ATG and everything but ... aside from an aging Norton the best he's beaten is ... Gerry Cooney

The closest Holmes ever fought to a swarmer of Frazier's caliber was Tyson ... Tyson wasn't actually a swarmer but he was similar in size to Frazier ... and Tyson mauled him EASY.

Frazier fought plenty of boxer types and beat near prime Ali