You can't always judge fitness by appearance. I'm 6' tall, weigh 260lbs (when I'm in shape, currently injured), and I'm always overweight (at least 15-20lbs). I can work all day (utility worker, lots and lots of digging), out work all my friends in the gym, do more push ups and sit ups than guy half my size, then wrestle with a fresh guy until he gets tired.

Some people just have a fat body type.