The Top-Down Reformer’s Calculation Problem

Two recent examples.

1. I was invited to attend the Progressive Policy Institute on Wednesday, but not as a speaker. The topic is introduced by saying

Now that Congress has passed the Every Student Succeeds Act (ESSA), states are revamping their federally required systems to measure school quality and hold schools accountable for performance. But most are doing so using outdated assumptions, holdovers from the Industrial Era, when cookie-cutter public schools followed orders from central headquarters and students were assigned to the closest school.

In today’s world, that is no longer the norm. We are migrating toward systems made up of diverse, fairly autonomous schools of choice, some of them operated by independent organizations, as charter, contract, or innovation schools. Before revising their measurement and accountability systems, states need to rethink their assumptions.

2. And David Cutler must be happy to read this story.

Medicare on Friday unveiled a far-reaching overhaul of how it pays doctors and other clinicians. Compensation for medical professionals will start taking into account the quality of service – not just quantity.

A Nobel Prize in economics was just awarded in part for the insight that it is a bad idea to compensate workers on factors that are heavily influenced by luck. In my view, having someone in Washington evaluate a school or a teacher or a doctor does exactly that.

People who are close to the schooling process, including parents, peers, and principals, can use judgment to evaluate teachers. That’s the way it used to work 50 years ago, before the advent of consolidated, unionized school districts.

For doctors, the prevalence of third-party payments means that their compensation is being determined by remote bureaucrats regardless.

The Case for Sticking with the Null Hypothesis

Jesse Singal writes,

As things continue to unfold, there will be at least some correlation between which areas of research get hit the hardest by replication issues and which areas of research offer the most optimistic accounts of human nature, potential, and malleability.

Pointer from Tyler Cowen.

Studies that show significant effects of educational interventions are right in this wheelhouse. That is why until they are scaled, replicated, and shown to have durable effects, you should view accounts of such studies with skepticism.

College Loan Default Not Related to High Tuition?

Jason Delisle says,

But if you get in the heads of people — and I did a focus group on this a year ago — you maybe went to school, it turns out it wasn’t for you or the school misled you, or you didn’t get a job in that field, and so you’re not really excited about paying back your $8,900 because you feel like you didn’t get anything for it.

Affluent people assume that student loan debt must be related to the high tuitions at top-tier schools with which they are familiar. DeLisle says that instead many of the defaults are occurring at lower-tier schools. The tuition and debt levels are low, but the students do not think they got anything out of the school, so they are reluctant to repay loans.

There is much more in this depressing interview. It is depressing how much of a discrepancy there is between a sensible policy on higher education and what we are actually likely to see.

Can Bad Nurture Make Things Worse?

Neerav Kingsland writes,

It’s very easy to see how a totally dysfunctional environment could negatively impact students, whereas it’s a little more difficult to tease out the additional impact on students once the basics are in place.

There may be exceptions to the null hypothesis on the extreme down side. That is, the very worst parents, the very worst teachers, and the very worst schools might have an adverse effect on young people that is causal, significant, long-lasting, and replicable.

The Null Hypothesis and Charter Schools

Will Dobbie and Roland G. Fryar write,

In this paper, we estimate the impact of charter schools on early-life labor market outcomes using administrative data from Texas. We find that, at the mean, charter schools have no impact on test scores and a negative impact on earnings. No Excuses charter schools increase test scores and four-year college enrollment, but have a small and statistically insignificant impact on earnings, while regular charter schools decrease test scores, four-year college enrollment, and earnings. Using school-level estimates, we find that charter schools that decrease test scores also tend to decrease earnings, while charter schools that increase test scores have no discernible impact on earnings.

The authors seem to think that their findings are precise and require some logical explanation. My guess is that their findings are random variations around the null hypothesis.

Hansonian Medicine

Although I have scheduled posts through the weekend, blogging might be light after that. A relative is struggling from an encounter with Hansonian medicine.

A robust finding in health care economics is that when you compare two populations with similar characteristics, the population on which more is spent on medical care enjoys no better outcomes, where outcomes are usually measured in terms of mortality. Given that we know that some treatments do work, this represents a puzzle.

The most radical way of resolving the puzzle is due to Robin Hanson. He suggests that the treatments that work are offset in the aggregate by treatments that cause harm. It is the latter that I have dubbed “Hansonian medicine.” I have myself witnessed Hansonian medicine take the lives of elderly relatives, although their lives were not shortened by much and their lives almost certainly had been prolonged by previous treatments.

The current episode concerns a recent procedure on a not-so-elderly relative that resulted in a severe infection. Moreover, the condition for which the procedure was undertaken is something that I always suspected may have been brought on by taking statin drugs, so that for years I have said no to doctor recommendations for me to take statins. (Just now, I googled and found that some recent research might support my hypothesis. However, I believe that the consensus is that my personal views are wrong and that statins are a low-cost, high-benefit treatment, which is the opposite of Hansonian medicine.)

Recent Pre-K Studies Not Optimistic

Lindsey Burke and Salim Furth write,

New studies of large-scale preschool programs in Quebec and Tennessee show that vastly expanding access to free or subsidized preschool may worsen behavioral and emotional outcomes.

You may have seen it claimed elsewhere that pre-K has been proven to work. Read the whole article. I am reminded of David Weinberger’s observation that nowadays for every fact there is an equal and opposite fact.

Meanwhile, long live the null hypothesis.

Question from a Commenter

He asks,

If we are over-educating our workforce, then why don’t entrepreneurs find and train non-college workers at lower lifetime salaries?

Very good question. Equivalently, why don’t non-college workers try to convince entrepreneurs that they can do the same work at lower pay?

Some possible answers:

1. The Caplan answer is that the non-college worker is attempting to work around the system, and thereby signaling a non-conformist personality that will be difficult to train and integrate into the firm.

2. The Cowen answer is that the college-educated individual is better able to deal with authority–knowing when to question and when to keep silent–and will therefore be more productive. In other words, college makes someone more productive even in a job that does not “require” a college education.

3. I would suggest another possible answer. Perhaps college-educated workers do not get higher salaries than non-college workers in the same jobs.* Perhaps the goal of attending college is to maintain or raise one’s social status. Getting a “college-appropriate” job would be nice, but it is not absolutely necessary in order to feel that you have raised your status, or at least made the attempt.

*Note: I believe that studies that have looked at this *do* show college-educated workers earning higher salaries, even in the same jobs. But I think it is hard to do this quasi-experimentally, which you would have to do in order to control for cognitive ability and conscientiousness.