My personal take is that there may be lots of muslims who are not personally violent, but they don't seem interested in standing up for the safety of non-muslims when the more vicious muslims show up and start acting up.
It would be like having 50 neighbors would never attack you but would also not do a thing to help you if 2 of your other neighbors decided to beat you to death.
If other groups of people are allowed to behave in ways based on in-group preferences, I don't see why we should not do the same when necessary.