I would normally agree with you as a matter of principle, but what if you were Jewish and your neighbour insisted on flying a Nazi swastika flag on a huge pole that you could see from your house?

Or if you were black, ditto with a National Front or KKK flag or something?

These things are not easy to make blanket statements about ......