In fact, we can carry your point further. You seem to be making the point that the right should be allowed to be harmful just for the sake of having opposing views be readily available and seen by all.
Let's say this carried over to more obvious harms. For example, the ultra-far-right thinks they should actively kill anyone not born a cis white male. Supposed society had secret exceptions to the rules and they were allowed to do this because they believe it? You'd allow it?