There is no reason for that to be happening, unless your timing is off. You don't touch the front of the glove, the part that makes contact when a punch lands. You make contact with his wrist area, swat the punch aside (misdirect it, is a better way to say it).