There's nothing holding you back. If you have a genuine interest in taking up boxing, then that's all you really need.

If you're having some cold feet about it, then go visit some gyms. If possible just pay for a few days worth of boxing and see what's there to like. Just keep in mind that the quality between boxing gyms/clubs vary, it's worth checking several gyms to find one that really suits you.