I think he'll only lose if he has more than the 2/3 fights he is talking about or tries going up in weight too high.....
As many have said i dont think the SMW and LHW divisions are strong enough to hand him a defeat at the moment. The most likely defeat would be by hopkins but i think Joe would eek out a UD against him.