Doubts about teacher value added

Marianne Bitner and others write,

Using administrative data from New York City, we find estimated teacher “effects” on height that are comparable in magnitude to actual teacher effects on math and ELA achievement, 0.22: compared to 0.29: and 0.26: respectively. On its face, such results raise concerns about the validity of these models.

. . .our results provide a cautionary tale for the naïve application of VAMs to teacher
evaluation and other settings. They point to the possibility of the misidentification of sizable teacher
“effects” where none exist. These effects may be due in part to spurious variation driven by the typically
small samples of children used to estimate a teacher’s individual effect.

VAMs = value-added measures. Pointer from a reader. I note that some recent NBER working papers are now free downloads. Others are not. This one is.

Lest you miss the point, this paper shows that the same methods that purport to show an effect of teachers on student achievement also show an effect of teachers on student height. But the effect of teachers on height is almost surely spurious. So the effect of teachers on achievement may also be spurious.

1. This provides vindication for Jerry Muller’s The Tyranny of Metrics.

2. It provides support for the Null Hypothesis.

3. The research that seemed to show a big effect of teachers (e.g., Raj Chetty on kindergarten teachers) got a lot of play in the press. But that had social desirability bias going for it. I would be surprised if this paper receives similar notice.

Contestable beliefs vs. sacred beliefs

In what way is polarization worse now than it was in the 1960s? I think that the answer has to do with contestable beliefs.

Most beliefs about human affairs should be treated as contestable. For example, would reducing the role of government in American health care result in a more cost-effective or less cost-effective system? I have my own opinion, and I hold it fairly strongly, but I see the issue as contestable.

1. A big reason that I prefer “beliefs about human affairs” to “social science” is that the term “science” sometimes suggests a process for arriving at certainty about issues that I suspect will remain unsettled, such as the issue about the role of government in American health care.

2. Note that my opinion that “most beliefs about human affairs should be treated as contestable” is itself a contestable belief. But I hold that opinion very strongly–more strongly than I hold my opinions about the health care system.

3. I might use the term “sacred beliefs” to describe strong opinions about human affairs that you don’t treat as contestable. For example, suppose that you believe that all adverse outcomes for African-Americans must be ascribed to discrimination and racism. Moreover, you see any questioning of that belief as racist. Then for you that belief is not contestable, and I would call it a sacred belief or quasi-religious conviction.

4. Suppose that you don’t believe that the Holocaust took place. I strongly disagree, and I question your reasonableness, but my belief that the Holocaust took place is in the contestable column, not in the sacred belief column.

5. When you move your opinions out of the contestable beliefs column and into the sacred beliefs column, a number of dangerous things happen. Because your beliefs are not contestable, your discourse no longer takes place in what I call Persuasion Mode. Instead, you turn to Demonization Mode. (See my Cato Unbound essay, Can We Improve Political Discourse?.) You become intolerant. You see disagreement as heresy, and you want to punish heretics.

6. As I remember it, the radicalism of the 1960s did not involve moving beliefs into the sacred beliefs column. In fact, it was more the opposite. There was a sacred belief that really horrible things would happen to us if we another country go Communist, and the anti-war movement challenged that regarding Vietnam, making the belief contestable. There was a sacred belief that if you were homosexual there was something wrong with you (the secular version of this was psychoanalytic), but that belief became contestable. Eventually, a lot of people changed their minds.

Cultures of conventional failure

In this essay, I will offer a theory of slow progress based on the maxim, “It is often easier to fail conventionally than to succeed unconventionally.” My claim will be that in health care, education, and construction/infrastructure, the consuming public is more receptive to conventional failure than unconventional success. This is less the case in other realms, so progress is faster in those realms. We are willing to try ride-sharing apps or airbnb, but no middle-class parent wants to tell their peers that their kids are not going to an established college.

In the Zuckerberg-Cowen-Collison conversation, Zuckerberg repeatedly asks why costs are high in health care, education, and rent.

So you were talking a minute ago about the explosion in costs in healthcare. And right now, I think one of the defining aspects of the moment that we’re in is a lot of the basic costs of living for a lot of people have just increased a lot. . .things that matter so much like healthcare, education, rent–those things have generally just increased, right? And the normal dynamics that you’d be hoping would play out aren’t. And to some degree, for the quality of life for a lot of people, the increases in those costs may even be dwarfing all the other advances in everything else

1. As I was listening, I was frustrated, because I wanted to point to my essay What Gets Expensive, and Why?. There I include the Baumol effect, but note my critical comments on the attempt by Alex Tabarrok to put it all on the Baumol effect.

2. Patrick notes that our ability to do large construction projects has declined over the past 70 years or so.

it’s very clear that our productivity has fallen off a cliff and for reasons that we can be pretty sure are not that it’s getting intrinsically harder. And so, for example, when New York decided to build the subway in 1900…4.7 years later, they opened 23 subway stations, and in 2019 dollars, they spent just over a billion dollars doing so. … When New York decided to build the Second Avenue subway in the year 2000, 17 years later they opened three stations and they spent $4 1/2 billion doing so. And so our productivity in subway construction has, at least in New York, decreased by a factor of 40. … California, you have high speed rail where… when France decided to build the TGV, its high speed rail, it opened the first line after five years. California started pursuing high speed rail 11 years ago. They forecast–we forecast–being finished in 2033. So we project a 25-year project, but of course, that’s a projection. It’ll probably end up being much longer

3. I already gave away my instinct on this when I wrote,

my inclination is to focus on broader cultural values. The enemies of progress are fear of novelty and envy of success. My thinking is that when those enemies hold sway, progress will be slow. When those enemies are weak, progress will be rapid.

I agree with Tyler that there is a lack of will. In the case of infrastructure projects or housing development, we now have a culture of conventional failure. Look at how hard it has been for Google to try to find a city that will allow it to experiment with a city of the future. Cities are only willing to approve politically correct development–bicycle lanes, as opposed to dedicated lanes for self-driving cars.

4. Patrick says,

empirically the entry costs of forming a new university are really high, but that’s not because there’s a kind of formal toll you have to pay. It’s not like zoning where there are deliberate, specific legal restrictions that prohibit you from doing so. But just as a practical matter sociologically, institutionally, accreditation dynamics, who knows, it’s apparently almost impossibly difficult to create a successful new university today.

Again, that is because we prefer conventional failure to unconventional success. I recently was hosted by a family in Texas. The oldest daughter was in the midst of applying to college. I wanted to scream “No! Don’t do it!” I do not believe that she is ready to go to college. I think she would be much better off just working at a low-paying job for a year or two and living on her own. I believe that is true for the vast majority of high school seniors these days.

Patrick is in the business of making it easier to start a company. Suppose he were in the business of making it easier to start a university. From the standpoint of technology, that seems like a very plausible business. Tools exist to deliver education in different ways. Look at Tyler’s and Alex’s MRU. The barriers are mostly cultural. Nobody wants to be the parent whose child succeeds unconventionally by taking a nontraditional approach to higher education.

I also want to scream “No!” when I see wealthy donors giving money to universities. The top schools have these enormous endowments already, which act like moats protecting them from competition. Don’t help them fill their moats! Instead, put that money into higher education start-ups. But if you give to your alma mater or to create a research institute at an established university, you can enjoy conventional failure. That seems to be more appealing to philanthropists than unconventional success.

Tyler and Patrick offer some provocative views about the way that success in research tends to come from less conventional institutional processes. But people stick with the conventional. For example, Tyler says

So I think in general, big questions are under-studied– the tenure system, I think, increasingly is broken. A lot of academics do work pretty hard, but that so much of your audience is a narrowly defined set of peers who write you reference and tenure letters–I think we need to change. And the incentive for academics to integrate with practitioners and learn from them and actually try doing things–we need more of that. I’ve often suggested for graduate school, instead of taking a class, everyone should be sent to a not-so-highincome village for two weeks. They can do whatever they want. Just go for two weeks, think about things. No one wants to do this. No one wants to experiment with it.

And I would add, require internships for economists. You can learn a lot while working in business.

Turning to health care, I think that Zuckerberg over-states the amount of money wasted in futile care in the last six months of life. But I think that the point is correct that we undergo many procedures with high costs and low benefits. I strongly believe that if we took away dollars at the margin from medical procedures and put those dollars into public health measures, the net effect would be positive. But wasting money on medical procedures with high costs and low benefits is a way to fail conventionally.

In short, when it comes to urban construction/civil engineering, education, and health care, we have evolved cultures of conventional failure. Innovation and entrepreneurship are discouraged. The heavy influence of government in these sectors probably serves to reinforce this. But ultimately the political process gives us something like what most people want. As Pogo would put it, we have met the enemy of progress, and he is us.

More from the Cato Unbound symposium

I write,

Unfortunately, we are seeing on campus a form of reductionist progressivism that actually does take the oppressor-oppressed axis as the sole basis for framing issues. The campus justice activists are not only subject to the psychology that inclines us away from Persuasion Mode and toward Demonization Mode. Their very ideology justifies Demonization of dead (and living) white males while treating the values of free speech and open inquiry as tools of oppression.

Jonathan Rauch writes,

Pace Kling, I don’t think “centralized curation of content” and content restrictions are most of what will happen as digital media grapple with antisocial behavior, though we’ll certainly see some of those (and have, and should). More important, and more successful, will be human-machine partnerships and platform redesigns which identify toxic behavior and content and help users avoid them.

There is more at both links. But again, I don’t think that bad actors and factually incorrect posts are the core problem. The core problem is the way otherwise good people behave on social media.

The problems with social media

by Jonathan Haidt. Self-recommending (although it seems to have annoyed Handle).

The problem may not be connectivity itself but rather the way social media turns so much communication into a public performance. We often think of communication as a two-way street. Intimacy builds as partners take turns, laugh at each other’s jokes, and make reciprocal disclosures. What happens, though, when grandstands are erected along both sides of that street and then filled with friends, acquaintances, rivals, and strangers, all passing judgment and offering commentary?

Read the whole thing.

Also, Haidt has done a two-minute animated video. I agree with some of its points, but it would not be my approach.