I think fighters can damage their legacy if they lose fights near their prime. The only way to know if a fighter was in their prime is when their career is over, and you can review it retrospectively. If Jones retired after Ruiz, Tyson after Spinks, their legacy would be higher than it is now.

Fighters like Duran, Holyfield and Holmes have had great moments long after their prime which can enhance their legacy. Duran beating Barclay, Holmes beating Mercer and Holyfield nearly beating Valuev (but that one may be stretching it a bit).