Big people that are athletic are pushed into sports. It's natural to assume most big athletes if not boxing are participating in other sports. It's possible many potential heavyweight fighters are doing other things. But sports is most likely.

Think of Lebron James. He's 6'8 and ripped with fast hands and feet and amazing coordination and explosiveness. Do you really think he couldn't have been a great boxer if he had grown up in the gym? I think it's highly unlikely that he wouldn't be great.

Then there is the perception your coaches give. My trainer wanted me to quit football and basketball. My football coach wanted me to quit boxing because I would drop so much weight. My basketball coach wanted me to quit football because my scholarship would be endangered by risk of injury. These coaches lead you to believe that you have to choose a sport. So we are programmed to believe some have chosen other sports.