The biggest risk of a Manfredo fight is "what if" Joe lost