Correct. A century of segregation doesn't count because in modern times that has been relegated to the dust heap of history.
That's how constitutional interpretation works.
If there is general consensus today, then there is a lack of controversy today, and that counts for a whole lot in the way these questions are addressed.
This is nothing new, this has been the norm for a long time.