Here is another spin on this. Has not fighting some big names made him a bigger name than what he may well have been had he fought the likes of RJJ, BHOP, etc.
If he'd had 3 or 4 defenses then fought RJJ and been Ko'd early he may have lost confidence and never been the champion and great fighter he is today. I'm not saying this is what would have happened in a fight between Roy and Joe, just a hypothetical.

If he'd fought Jones or Hopkins in say 2000 they were probably more on top of their game than what Joe was then, so by not fighting those guys he may have done himself the favor.

Or maybe im talking shit.