Manny was an awesome fighter during that period and Floyd conveniantly avoided him during that time. If Floyd fought him now it would suggest he knows now he could beat him.

Floyd has grown into the welterweight division and is a better fighter than when he beat Baldimor. So there was a small chance Manny could have beaten Floyd.