The fact it is hard for Joe to become P4P number 1 can be attributed a lot to the fact that it took so long for him to fight tough opposition. I mean if he'd fought and beat younger versions of RJJ and BHOP then i think he'd be number 1 now. Instead he was rematching Veit and the like.

I agree there isnt the talent around between 160 and 175 at the moment to really help Joes legacy but 5-10 years ago there was plenty.
Younger RJJ
Younger BHOP
Darius Michalzcewski
Younger Tarver
Younger Glen Johnson
to name a few.
I think Joe has the talent, work ethic and desire to be the P4P king but it wont happen because he has left his run too late.