Where do people get this idea from? Do they not realise that De La Hoya-Mayweather brought in more money than any other fight in history, and it was all of about 5 seconds ago in the grand scheme of things? Granted the heavyweight division is absent of a true superstar and attractive personality, but that doesn't mean anything in regards to the sport.
A lot of times their argument is the growing popularity of MMA, but I tell those people time and time again that while MMA will continue to be popular, it'll likely never achieve the status of boxing. The reason being is that boxing is something that's become an iconic sport all around the world, a universal medium.
Also, I usually find that it's Americans making these claims more times than not, and I think that has more to do with the absence of a dominant American heavyweight/icon than anything going on in the sport.