Its harder to pull your arm back, there is nothing for your bicep to contract against so more effort to bring the punch back.

The bag/pads/kids head/ribs stop your elbow joint from opening 100% so your biceps don't have to work as hard, the same goes for all the other joints and muscle groups used when punching, thats why missing tired you out more.