Sounds like it could be common muscle fatigue. How are you holding your arms? If you are holding them unnaturally high they will tire much faster. How do you feel like on a fresh day? How much are you training? some guys can't do 5-6 days a week because they are not in the condition to do so. If you could tell us if there's any particular pains/soreness we could tell you how you can adjust your stance, or the way you throw your punches.

There's a couple good barbell/dumbell exercises that come mind when increasing shoulder/arm endurance. But first I'd like to know how your arms feel when you are working the speedbag if you do so.