I thought Duddy's facial injuries was the reason he didn't go through with the Pavlik fight?

Clearly his mob aint complete sickos.