Not necessarily, look at Eddy Futch's fighters, or some of Freddy Roach's. Not all of them are great at it but average fighter can do it too. Hell, even Freddy Roach wasn't a world beater but he learned those moves too. Look at the shoulder-roll counter that Futch taught most of his fighters. One of them was dropping a left lead to draw right hand counter, the punch can be parried or blocked with the left shoulder, and a right hand, or flicker jab can be thrown in retaliation.

The average guy will get hit the first few times as you may have faced, but naturally as you know there's timing in landing a punch, there's timing in defense as well. Unless you've fine-tuned it down to the feeling and reflex of it, I'll agree that learning boxer is going to be pretty reserved about it. Even James Toney was a big gym rat when he was learning these moves, if you want to get it down you have to work it into your sparring/bag-work and especially your sparring sessions.