Its got nothing to do with colour.

Joe Calzaghe is disliked by loads of people on this site, he's white.

Antonio Tarver is disliked equally, he's black.

Audley Harrisons yellow and he's disliked by all!