I also think the reason why he is more physically sound than everyone is that half of his training doesn't really revolve around "What is my opponent going to do?" Floyd trains for Floyd. He doesn't have to formulate game plans from scratch because of his ability to change game plans mid-fight. So he has more time to physically prepare.