It all comes down to moderation really. If once a month you smoke a joint with some buddies, you won't be worse off then someone who doesn't smoke at all. It will have no effect on your training whatsoever.

Make it a habit and see what happens. The same would go for making drinking a habit. Drinking may actually be worse because it may also put the pounds on.