The Null Hypothesis, Compounded

Posted on by

In the WaPo, Lindsey Layton reports

A new study of 10,000 teachers found that professional development — the teacher workshops and training that cost taxpayers billions of dollars each year — is largely a waste.

The Null Hypothesis is this:

1. Take any educational intervention
2. Conduct a rigorous controlled experiment.
3. Look for results that do not fade out within a year or two.
4. If you find apparent success, try to replicate it
5. You will not find significant effects in all of steps (1) – (4)

For teacher training to work, you have to deal with the null hypothesis at two layers. First, you have to find an intervention that significantly affects teachers. Then that intervention has to significantly effect students. This is the null hypothesis compounded. In Building a Better Teacher, education journalist Elizabeth Green claims that some methods of teacher training survive (1) – (3), but I remained skeptical of (4). Green was interviewed by Russ Roberts in an econtalk episode about a year ago.

10 thoughts on “The Null Hypothesis, Compounded”

  1. Of course it doesn’t. Nor do the graduate education classes that teachers take in masters programs. But the training gets them out of the classroom, can be used to justify salary increases, and provides the expected trappings of professionalism — sort of ‘professionalism theater’. Teacher training programs also provide a lot of employment in the education-industrial complex for administrators, trainers, and consultants.

  3. >> 4. If you find apparent success, try to replicate it

    Replicating the same social results across time in most cases mandates employment of different methods. Social science always endures the problem of identifying the correct premise.

  4. Arnold – you don’t think Raj Chetty’s AER paper (with Friedman and Rockoff) doesn’t meet all of these?

  5. “Nor do the graduate education classes that teachers take in masters programs. But the training gets them out of the classroom, can be used to justify salary increases, and provides the expected trappings of professionalism — sort of ‘professionalism theater’.

    True that graduate training makes no difference. But everything else you say in this sentence is completely incorrect. Teachers don’t get out of class for any education they take on their own–nor do they get reimbursed for it. In many districts, it doesn’t have to be a master’s program. It can be any education at all. And the additional education is not “used to justify” additional pay. No justification is necessary. It’s a one for one pay bump. Credits = bump in the vast majority of districts. Nor does anyone pretend that giving credits gives the appearance of professionalism. It’s how teachers get more money on the scale. Full stop. No romanticization, no rationale, no pretense. Teachers pay money to get education, they get more pay. Many of them double it up–get an administrative credential. They get more money on the scale, and get the chance of moving into administration.

    But before you yammer on about how stupid it is to give teachers more money simply for getting more education when it doesn’t make any difference, remember Arnold’s Null Hypothesis. Nothing makes any difference. So there’s no point in merit pay or any other form of giving teachers more money. Teachers don’t work overtime, like cops, and they don’t get moved up different ranks (patrol level 1, or whatever cops do).

    So this is just a way to give teachers more money. Much cheaper than promotions, since a lot of them just don’t have time to go to school in the early days.

    “Teacher training programs also provide a lot of employment in the education-industrial complex for administrators, trainers, and consultants”

    That’s not “also”. That’s *the* reason. As I’ve just pointed out, we don’t get any money for PD. The overwhelming majority of teachers consider it a waste of time. It’s a face-saving exercise for administrators and districts to shut up the various education reformers, most of whom run consulting businesses or know someone with a consulting business.

    And Chetty’s research is a total joke. I’m sure it’s professionally done. It’s just ludicrous.

    • “But before you yammer on about how stupid it is to give teachers more money simply for getting more education when it doesn’t make any difference, remember Arnold’s Null Hypothesis. Nothing makes any difference.”

      That isn’t exactly why I think it is stupid. If anything could make a differerence, I would trust teachers find it. But why would they look if we pay them not to?

    • “Teachers don’t get out of class for any education they take on their own–nor do they get reimbursed for it.”

      Teachers get paid during ‘in-service’ training days and — as you point out — they ultimately get paid for continuing education classes by pay scale increases.

      “And the additional education is not “used to justify” additional pay. No justification is necessary. It’s a one for one pay bump. Credits = bump in the vast majority of districts.”

      But the justification for the pay bumps is that the continuing education makes teachers better (what other justification/rationale could there be?). If you asked a superintendent or teacher union official why it makes sense to pay more for teachers with graduate education, would they say A) “Because continuing education is important for improving the quality of our teachers” or B) “Because we need some pretext for paying more over time and this is as good as any”? They might privately think (B), but they sure as hell aren’t going to say it in public.

      “But before you yammer on about how stupid it is to give teachers more money simply for getting more education when it doesn’t make any difference, remember Arnold’s Null Hypothesis. Nothing makes any difference. So there’s no point in merit pay or any other form of giving teachers more money.”

      The null hypothesis is that interventions don’t replicate and scale, NOT that there are no differences in effectiveness between teachers or that individual teachers never become more (or less) effective over time.

      “So this is just a way to give teachers more money. Much cheaper than promotions, since a lot of them just don’t have time to go to school in the early days.”

      But why on earth would we want an arbitrary way to give teachers more money–as opposed to paying highly effective teachers more than mediocre ones? We don’t need to know HOW to manufacture highly effective teachers, we just need ways to recognize and reward them. And such a system doesn’t have to be perfect. Measuring effectiveness of employees is difficult and imperfect in almost all industries, but organizations do it all the same.

Comments are closed.