IMO it's stamina rather than the loss of speed will be the problem. Carry more muscle and you won't last as long. The problem will start when an opponent takes his single shots fluch and doesn't blink, he might be able to outbox the cumbersome guys for a few rounds but they'll catch up to him.