Null Hypothesis watch

Eric A. Hanushek, Paul E. Peterson, Laura M. Talpey and Ludger Woessmann write,

We find that the socioeconomic achievement gap among the 1950s birth cohorts is very large—about 1.0 standard deviations between those in the top and bottom deciles of the socioeconomic distribution (the “90–10 gap”) and around 0.8 standard deviations between those in the top and bottom quartiles (the “75–25 gap”). These are very extensive disparities, as 1 standard deviation is approximately the difference in the average performance of students in 4th and 8th grades, or four years’ worth of learning. But though these inequalities are large, they have neither increased nor decreased significantly over the past 50 years.

It could be, however, that the picture is not as dismal as suggested. If overall changes in society, coupled with policy initiatives, have proportionately lifted all boats at the same rate, everybody might be better-off, even if gaps have not significantly changed. Using the same data as for the gap analysis, we find gains in average student performance of about 0.5 standard deviations for students at age 14, or roughly 0.1 standard deviations per decade. But, surprisingly, over the last quarter century, those gains disappear for students by age 17. In other words, there is no rising tide for students as they leave school for college and careers.

The lack of progress is not due to lack of spending. In an op-ed piece, two of the authors point out that

Since 1980 the federal government has spent almost $500 billion (in 2017 dollars) on compensatory education and another $250 billion on Head Start programs for low-income preschoolers. Forty-five states, acting under court orders, threats or settlements, have directed money specifically to their neediest districts.

And overall, of course, inflation-adjusted spending per pupil in the U.S. more than doubled between 1970 and 2006.

I should note that the authors themselves are not devotees of the null hypothesis. Instead, they call for more efforts to improve teacher quality and to address the issue of “fade-out” of achievement gains between age 14 and age 17.

12 thoughts on “Null Hypothesis watch

    • The typical NAEP trend narrative for years: “Impressive gains in elementary and middle, but not high school.”

      But are those “gains” in any meaningful sense? Or just: arriving at same point a little earlier in life.

      Because of the USA standards movement, topics like “how to add fractions” are taught sooner, often a grade or two earlier than years ago.

      So it creates some easy wins for American kids on NAEP, But then that illusory advantage disappears in hs. So it’s not true fade out…

  1. The overlap in distributions among 4-8 grade levels is the very definition of Null hypothesis.

  2. There are important intelligence changes that happen during puberty that nobody seems able to overcome. It makes sense that puberty would be a point of evolutionary pressure.

  3. Around puberty, young people cease being “information sponges”. They get new interests, e.g., possible romantic/sexual partners, and become more interested in questions like, “what is my place in the world?”, “what will my future be like?” At the same time, school becomes more academic, more removed from most young people’s concerns.

  4. “they call for more efforts to improve teacher quality”

    Well that is not going to happen with the ed schools that the United States has or with the quality of students getting ed degrees which is notoriously low and the supply for which is pointlessly restricted by licensing requirements that are completely unrelated to job performance and outcomes. Illiteracy is not unknown among practicing US teachers.

    The US might learn how to improve by looking to up to countries with mediocre or better education systems. Parents should learn from the Asian leaders in education by investing in better supplemental education programs. Rather than just paying for test prep courses, parents need to realize that the schools will never provide the education that their children need. They need to look to after-school educational programs with hard core learning activities like in the cram school.

    Policy makers could learn too. World happiness leader Finland has great schools because it has intelligent teachers. To get smarter teachers into the US classrooms, a special immigration category should be created for high IQ individuals who agree to teach in US classrooms under a pay for performance outcomes based compensation arrangement. Call it an IQ Lottery. Offer an IQ test and with admission to lets say everybody scoring over 125. To get around the local unions protecting the current incumbents from competition, perhaps a Federal Teachers Corps program along the lines of Teach for America could be created with high IQ immigrants being paid by the feds to fill teaching positions in local schools that would otherwise go to US ed school graduates. It would be a good way to have the fed preempt state and local teaching license requirements. And a couple million new citizens from places like Nigeria, India, Korea, Ukraine, Poland and elsewhere would boost diversity, bring fresh and creative perspectives, and help build closer international ties, as well as improve educational outcomes. Funding could come from eliminating the rest of the Department of Education and creating a new $10,000 per person processing fee for immigrants who enter the country unlawfully. The enormous backlog of illegal aliens awaiting processing might dry up a little as an added bonus. In many ways a triple boost to the economy.

    • This is the sort of idiotic crap that screams “you will do stupid things if you don’t believe the Null Hypothesis”.

      Asia and Finland do well because of genetics. I doubt their teachers or teaching method are substantially better. Asian schools in particular are criticized for mindless grinding mediocrity even by Asians.

      And why would you want your smartest becoming teachers? Does it take a genius to babysit children? Do most of these children have some grand potential that only a genius can bring out? Most kids are mediocrities that will be taught by mediocrities. That’s the most effective system. You need your geniuses out there building society. If you want to single out your next generation of geniuses and give them special attention then that passes an ROI test, but today we do the opposite (shower resources on dead ends). And of course we measure our school and teacher performance based on the poor performance of our minorities, which they can’t really change.

    • You assume that there is a known way to “improve teacher quality”. There isn’t. Full stop. The people who run ed schools don’t know it. The people who run foreign school systems don’t know it.

      Getting smarter people into teaching doesn’t help students. It doesn’t “improve teacher quality” if that means students learning more. Studies can’t find smarter teachers making a difference. And one wouldn’t expect it to. Smart people learn easily. Most students don’t. Smart people who want to be good teachers have to get out of themselves, have to unlearn how it was for them.

      To be clear, there have to be minimum standards for teachers. They should be literate and know the material, which implies at least an average level of smarts. And those teaching academically challenging courses (Honors Physics?) should be pretty bright and be able to answer questions that go beyond the text book.

      • Largely agree.

        There is a small teacher residency and graduate school in Boston, called Match, whose grads do perform a standard deviation above average in generating student gains. But we have not been able to do it as a randomized controlled trial. We can’t easily separate selection effects, training effects, and school effects.

        A few other small programs around the USA have similar gains.

  5. Maybe the money is not being spent the “right way” — directly on the students.
    Give money to students who show up without discipline problems. ($2 / day?)
    Give money to those who show effort? Doing book reports, etc.
    Give money for good scores?

    Are there any school programs that pay students to show up, be good, and do good work? Is it illegal?

    It should be tried.

  6. Maybe I misunderstand but the referenced research says it compares the “socioeconomic achievement gap” with “socioeconomic distribution” across a 50 year time. Okay, going out on a limb here since it is 40 years since I studied statistics but the “90-10 gap” and “75-25 gap” are simply measures of data distribution, as is standard deviation which seems the reference. By using standard deviation, I guess they have demonstrated that the data sets can be considered normal. But if the data sets are normal, why wouldn’t it be the case that even a billion years from now the data on the “90-10” gap would translate to a constant standard deviation? I must misunderstand this somehow.

Comments are closed.