It will improve your speed and power, dont go overboard with the weight though, max 5lbs (per hand) IMO.

It doesnt have to just be punches, you can do any dumbell drill, 30 second bursts and then immediately on to the next one, 6 excercises = 1 round, great stuff for muscle endurance.