If you're unmotivated then it's likely the guy is going for you for no real reason, hence the guy is a complete turd. It's hard to walk away as you'll give him confidence by doing this and he might attack you. Some people are good at bluffing and can send the fear right back into the other guy without even touching him, probably the best way to do this is to appear confident yourself in your abilities, which is itself easier to do when you are genuinely confident. Anyway, not to sure what my point is
p.s. there's no shame in walking away, you'll be better for it, of course defend yourself if someone attacks you.