Going back to Handle’s comment on problems for libertarian thought, he writes,
obsession with explicit state / government action and insouciant attitude regarding social pressures, when, in the modern era, the latter may have emerged as an even worse threat to the exercise of traditional liberties.
The ‘local freedom to coerce’ problem. If we are trying to increase welfare by giving people what they desire, we have to recognize that one of the things people desire is ‘a community’ and for their communities to have particular characters and sets of norms. There are certain forms of social experience or community life which are impossible to coordinate if the overall enterprise is deprived of some of the core, and at least mildly coercive, attributes of sovereignty.
On the first point, John Stuart Mill also worried about social pressure as a restriction on liberty. And on the second point, Barry Goldwater and Milton Friedman (at least if I remember correctly the relevant passages in Capitalism and Freedom) were against Federal intervention to protect African-Americans from segregation, even segregation imposed by state and local governments.
So these are longstanding problems for libertarians. My own position is that the best way to deal with social opprobrium or discrimination is to give the people who are hurt by those phenomena as much opportunity to exit as possible. I think that once you construe it as a problem that government must solve (by passing Civil Rights laws or regulating organizations) the overall consequences are likely to be worse than letting the problem be resolved through exit.
I hasten to add that exit is not a solution to every problem. Cities, in particular, are bundles of externalities. For any individual, some of these externalities are positive, and some of them are negative. If the positive externalities are strong enough, you will stay in a city and put up with major negative externalities. In theory, using government to get rid of those negative externalities would be an improvement. In practice, I have to say that it is the local government that is the negative externality where I live. That is, if you ask me what would motivate me to move, the first thing that comes to mind is the local government, which increasingly is going to collect taxes to pay for union pensions, not to provide actual services.