The reason these discussions often break down right about here is, IMO, because they have different working definitions of the concept of tolerance itself.

For example, the social contract comment above assumes an active definition like recognizing others’ personal sovereignty, i.e. their right to act and not be acted upon. Here one could imagine people as a group of countries with a multinational peace treaty. For one to be intolerant would be to break the ceasefire and attack another. Neighboring countries are within their right to defend themselves or punish this breach without violating the treaty themselves.

Another popular notion of tolerance would be better-termed “permissiveness,” as it amounts to passive affirmation of others’ value systems. This is the “you’re good, I’m good, we’re all good,” kumbaya definition of tolerance conservative talking heads prefer for numerous reasons.