There's no easy answer or blanket statement that can apply because athleticism is hard to quantify to any accurate degree when you're talking about boxing.

I think we've seen that you can only go so far with low levels of either one. Take a guy like Seth Mitchell, who is a fantastic athlete but not much of a boxer. Take Freddie Roach, who was a fantastic boxer but had no noticable athletic ability.