Part of the reason for me is the activity levels of the sport. The top names fight once or twice a year unlike in Golf, Tennis, Football (soccer!), where they play more regularly. In boxing you have to wait for the big fights where in other sports the top names are in action nearly every other week. With boxing you're left watching the up and coming fighters normally handing a beating to some poor sod who is in the ring to earn some cash, this only holds the attention for so long.

In addition for the neutral, fights like Gatti/Mayweather and Hopkins/Taylor do not do much for the sport. The first was a mismatch that if your not a big fan you would call boring, and the second again was boring for the first 8 rounds. If you only watch the sport for the big fights is it any wonder that you're left disattisfied after fights like these.