How much does eating poorly once in a while set you back?

[CFH]

Ok, I'm certain I already know the answer to this question, but I wanted to ask you guys anyways.

I've been going to the gym 5 - 6 days a week for a month and a half. 5 days a week, I do about an hour of intense cardio and as of a week ago I have twice a week added about a 45 min session of weight training. I'm not really trying to bulk up or anything, I'd just like to lose some weight and improve my health and endurance to a high level. I have also dramatically improved my diet and I now eat quite healthy the majority of the time.

Now for the meat of this post, on Saturday nights I have been allowing myself to cheat on my healthy eating. I usually drink some beer or liquor and I don't eat as well as I do the other days of the week. I'll usually have some fries at the bar, some chips, and a brioche and stuff like that.

Now my question is, how much is this hurting all the hard work I have put in throughout the rest of the week? I'm thinking of reducing my "cheat days" as I think of them to one or two days a month, is that better or should I just try to avoid them altogether? I must admit, I do like to drink about once every week or two, but I'll severely curtail that if I must.

Thanks for the help.