Well getting beyond the obvious fact that James SHOULD NOT be licensed to fight anywhere as it's been proven that he has suffered brain damage to the extent where his hormones are fucked up, at this point its a lose-lose for anyone in the top 10 to fight him, considering...

a) If you win, so what? You beat a 44 year old blown up middleweight.
b) He's not a guy that people look good against, even in winning.
c) If you lose to him, you can kiss your world title contention goodbye.