You can think of the quad graph as being a combination of the original planar graph with its planar dual.
Now, a (spherical) polyhedron is self-dual if and only if it’s isomorphic to its dual, or equivalently iff there is a part-reversing automorphism of its quad graph.
So that’s what I mean when I say that self-duality is akin to symmetry. Precisely, it is a symmetry of the quad graph. Since our original polyhedra(l graphs) have no symmetry, the only symmetries of the quad graph there could possibly be are part-reversing ones, i.e. dualities.