Yes, I had exactly the same arguments in a conversation yesterday. Those who cannot work from home benefit from less traffic jams and less crowded public transportation.
And what stops them from applying for jobs that can be done from home, like call center work?