it's both. it's very much patriarchy and as a woman, you definitely get an extra dose, but it's all about fear. patriarchy fears that they aren't enough, so use every means they can to falsely elevate themselves and shove everyone else down.
this includes fashion. they don't know if what they like is OK, so they go with a strict formula and hold everyone to it. suits. navy or dark grey. worsted wool. white or light blue shirt. red or blue tie. my uncle lived in NYC, was an artist. he had a trip to indiana, wore very trendy blazer and slacks. every person in the office he visited was in the "uniform". navy suits everywhere. if he'd been an employee, not a contractor, i'm sure they would have told him to change, like you were.
you still see every politician these days in the same uniform. why do you think they totally freaked out at obama's tan suit or hilary clinton wearing pant suits? they weren't following the rules.
small petty people hate anyone who has opinions and tastes and doesn't live in fear like them, so they lash out.