Null hypothesis watch

Posted on by

Sarah Cohodes and co-authors write,

We study a policy reform that allowed effective charter schools in Boston, Massachusetts to replicate their school models at new locations. Estimates based on randomized admission lotteries show that replication charter schools generate large achievement gains on par with those produced by their parent campuses. The average effectiveness of Boston’s charter middle school sector increased after the reform despite a doubling of charter market share. An exploration of mechanisms shows that Boston charter schools reduce the returns to teacher experience and compress the distribution of teacher effectiveness, suggesting the highly standardized practices in place at charter schools may facilitate replicability.

One of the main pillars of the null hypothesis is that education interventions that succeed as experiments often fail to replicate. If this study is valid (if it replicates, so to speak) then it helps knock down one of those pillars. Since it favors charter schools, I am actually rooting against the null hypothesis in this case. But we’ll see.

14 thoughts on “Null hypothesis watch”

  2. For a state-of-the-art analysis of problems in scaling of social experiments, see Omar A. Ubaydli, John List, and Dana Suskind, “The Science of Using Science: Towards an Understanding of the Threats to Scaling Experiments,” Working Paper 25848 (NBER, May 2019).

    Here is the abstract:
    “Policymakers are increasingly turning to insights gained from the experimental method as a means of informing public policies. Whether—and to what extent—insights from a research study scale to the level of the broader public is, in many situations, based on blind faith. This scale-up problem can lead to a vast waste of resources, a missed opportunity to improve people’s lives, and a diminution in the public’s trust in the scientific method’s ability to contribute to policymaking. This study provides a theoretical lens to deepen our understanding of the science of how to use science. Through a simple model, we highlight three elements of the scale-up problem: (1) when does evidence become actionable (appropriate statistical inference); (2) properties of the population; and (3) properties of the situation. We argue that until these three areas are fully understood and recognized by researchers and policymakers, the threats to scalability will render any scaling exercise as particularly vulnerable. In this way, our work represents a challenge to empiricists to estimate the nature and extent of how important the various threats to scalability are in practice, and to implement those in their original research.”

    And here is a link to an ungated version of the paper:
    http://s3.amazonaws.com/fieldexperiments-papers2/papers/00670.pdf

  3. Let us assume that teaching to standard tests is the same everywhere. We test the theory in a controlled situation and find out that ‘teaching to the test’ does not replicate. What would we assume? I am having a hard time with identifying the Null.

    The standard test may be badly defined, a separate issue. But we only allow standardized tests, hence replication is the Null, by definition of standard.

  4. I don’t really want to pay $5 for a paper, so could someone who has access to this describe what “large achievement gains” specifically means?

    I generally find these debates to be dismal because there is no coherent consensus on what we expect k-12 schools to be responsible for, much less any consensus on how we can measure a school’s performance. Until those ideas are well understood, I have very little faith in debates about strategy.

  5. I’d suggest that all “new” charter schools enjoy the benefits of a “new” system. I recall a paper that tested different lighting in different offices of the same company.
    One had brighter lights.
    One had dimmer lights.
    Most were control group, no changes.

    Both “changed” groups had better results. I had read similar results, so I tend to believe its conclusions — “any” identifiable change creates a burst of positive change. (I don’t think the study was long term to test how long the effect lasted; but it might be I just don’t remember).

    If parents have to choose a charter or not, the fact that they are choosing is certain to increase their involvement in their kids’ performance. That’s good. Possibly a huge advantage, tho probably merely a small advantage, of charter schools.

    I’m also rooting for the charter schools, but it’s important to follow the evidence. It’s also good to get more experience.

    It’s clear that current gov’t K-12 is giving bad results for poor folk, especially poor blacks.

  6. I am going to go with the Null Hypothesis mostly because measuring and changing education is so hard in general. (And a school improvement tend to a change in the student body not the teaching.) And the school district my kids go to the reputation of the Charter when it opened in 2009 was HUGE and wait list and now it is sort of “Meh” in general. And from I can tell the primary reason why is the classes are a lot more limited than the basic Public Education High School where some students can take a wider variety of classes. (Like second year Calculus.)

    Honestly, I wish
    1) The basics of education have not moved a whole lot and most parents want their kids to good education.
    2) Honestly, learning to navigate an uncaring bureaucracy is probably training for modern adult life as there is!
    3) Most educators underestimate the need for convenience here.
    4) Probably the biggest miss I see at my kids High School is more vocational avenues for careers. However, I wish the private sector would participate here.

    5) It could be my area but I find students that go to private or charter schools tend to the ones most full of themselves and the ones most likely to SJW.

    • And one aspect I don’t see from conservatives on charter schools or voucher systems, is how to get the best public school teachers to stop supporting teacher union and traditional public schools.

      • Why is it that the states with the strongest or most effective teachers unions, like Massachusetts, get better results and states with weak, ineffective unions, like Mississippi, get poorer results?

        Once you can explain this I will listen to your your anti-union biases.

        • asdf is right that the main driver of student test scores are students ability.

          And I have seen in my area, that the best teachers are the strongest union supporters and the strongest unions are in the ‘better off districts’. So my family is an good district and there is some union support while the great districts have much stronger unions.

          So I believe this is ‘circular’ function here:
          1) Teachers are increasingly crossing state borders so our Cali area is bringing some of the best potential Red state teachers. (Lots from Kansas here)
          2) The Better the district, the more the parents enforce school discipline and work ethic on their kids.
          3) These teachers work well with local parents. (which increases parent support for the union.)

          So Massachusetts the local system is working and parents support the union as part of the system whereas Mississippi has the problem that nobody thinks schools work so the union must be bad. Which also limits the interest of potential teachers in Mississippi, some of whom go private sector and others may move to another state.

          So again, I do believe a big barrier to private schools and vouchers, is teachers are a stakeholder and I failed to see a optimal school system is going to both lower teacher salaries and improve education long term. And doing away with the union, how do you convince the best teachers to reject the current system.

  7. Boston restricts charters, and Boston Public Schools are particularly hideous, and this is only middle schools. Just putting kids in a quiet environment will get a small boost of a test score. And the increase is only in math, and only .18 to .32 of a standard deviation, which is about what you’d expect with ruthless discipline that boots out noncompliers, puts them in school a lot longer, and has little in the way of electives.

    It’s absolutely nothing to do with teachers, scripts, curriculum, or principals.

    • Seems like a pretty good summary of the “look at my study that disproves everything” phenomenon. For a lot of broad social realities, like the null hypothesis, I used to try to hunt down data that would overturn it. And I used to read studies that refuted it people linked (and I suppose I still might from time to time if its an easy enough study to digest and I have free time).

      But for the most part its always the same. The study doesn’t show what it says it shows, or what it shows is a tiny effect, or what it shows isn’t meaningful to the broader question/context its being applied to. Eventually, you just stop caring when someone links a study. You know how it’s going to turn out. You’ve essentially arrived at “the answer” after reading so much and considering so much.

Comments are closed.