A reader asked for this.
The term “null hypothesis” comes from statistics. The word “null” means “no effect” and the null hypothesis is that an intervention has no effect on the outcome. If you were testing the effectiveness of a drug, the null hypothesis would be that the drug works no better than a placebo. If you do a study and you do find that the drug works better than the placebo, and this is not likely just an accident, then you reject the null hypothesis.
I apply the term “null hypothesis” in the context of education. My observation is that most of the time when an intervention in education is evaluated rigorously, it has no effect compared to a control group, or the effect wears out over time, or the effect cannot be duplicate in repeated experiments or at large scale.
I appreciate the clarification of your use of the term; because it points to some key issues. External validity, generalization, and regression to the mean.
The first one (external validity) says that you could see an effect under some very specific conditions, repeatedly…
but that generalization might not work the way you expect.
…and this is frequently seen in time series via ‘regression to the mean’ – an effect that should hypothetically persist (‘the world has changed!’) seems to fade because there was something else going on.
There’s a lot to be gained from reading scholars and bloggers who argue for the null hypothesis in education. Such individuals are skeptical of the benefits of providing the public with more subsidized education, or forcing more education on youngsters who are still under adult supervision / custody.
Many people have to be exposed to this idea repeatedly before it sounds plausible–just as they have to be exposed to the “Efficient Market Hypothesis” a few times before they grow to distrust all overconfident stock analysts.
Off the top of my head (and without consulting them) the following educational skeptics come to mind.
1. Prof. Arnold Kling, here. Especially under the tags “Economics of Education”
2. Bruce Charlton (search for his essays on, to paraphrase, ‘using psychometrics and skipping higher education entirely,’ back in his “Medical Hypotheses” days). As I recall, he has at least one essay on how his views of the topic changed as he learned more about psychometrics.
3. Greg Cochrane at West Hunter (see, for example, his recent reviews of Caplan’s _Against education_, but more generally his skepticism pervades the blog).
4. Caplan himself who has a new book on the topic.
5. The anonymous blogger who uses the pseudonym “Education Realist” and teaches high school math.
6. Slate Star Codex sometimes discusses the issue, especially with his maxim “If there’s any possible way it can be selection bias, it’s selection bias.”
7. Steve Sailer sometimes discusses the issue among his wide ranging writings
= – = – = – =
I remain agnostic–probably much of the benefit of education comes from assisting or forcing habits of self-control and self-regulation, which some people could get from other activities such as sports, work, military training, artistic performance, etc.
Peer effects are important. If school makes you respect people who work hard in school and push themselves to learn, it makes a difference. This is one of the points that Judith Rich Harris has made.
Personally, when I think of the topics that I would not have learned except in school, it’s heavily weighted toward mathematical / analytical fields–all my math, basically, plus programming and similar topics. A lot of the issue is “time on task.” For me, only being in a math class tends to promote adequate “time on task.”
To put it more simply, I could have learned most history on my own–and maybe even economics. I could not or would not have learned algebra without sitting in algebra class.
I’ve reached this conclusion about my own education conditional upon my own personal and family background: parents of high native intelligence, upper middle class milieu, my parents are good readers, they are adequately organized (not incapacitated by drugs or mental illness) I grew up in a house full of books with a blizzard of paper that flowed through it.
Someone from a different socio-economic class might have a very different experience. This reminds me of a point made repeatedly by E. D. Hirsch: it’s the lower economic and social strata who most need a decent classroom curriculum, because they cannot get the same sorts of educational inputs from peers or the home environment.
Citation for Hirsch: _The schools we need–and why we don’t have them_, circa 1996.
You might even say there’s something like a placebo effect in education. The placebo effect in medicine is mostly caused by the fact that the body tends to heal itself over time (this doesn’t mean every person recovers from an illness or injury but that the average person’s condition tends to improve). For a treatment to be effective, it needs to be better than the body’s innate healing ability. I think you could apply the same reasoning to education. People’s knowledge and skills tend to grow over time, especially when young (again, this doesn’t mean that every person is always learning but that the average person will have more knowledge and skills after a given span of time than they did before). So, the way you could think about it is, for an educational intervention to be effective, it needs to be significantly better than just what people will learn by greater experience or their own efforts due to their natural curiosity.
Charles’ comment is correct that all those mentioned are education skeptics. However, they don’t all agree.
For example, I flatly disagree with Caplan’s book and his approach in general. Cochrane thinks preschool for poor children is a waste of money, whereas I think the mistake lies in arguing that preschool improves academic outcomes. Who cares if it does? Get poor kids away from their often incompetent mothers and crushing environments. Give the kids some sense that the government cares about them.
SlateStarCodex is pro-charter, which I think is a terrible idea. Charters are basically private schools on the public dime, which means they can cap and skim. Both will increase the costs of education. Charters can only increase the amount we spend on education, which is crazy.
Caplan and SSC would certainly give free public education to all immigrants, as we do now, while I think we should limit free public education to citizens.
And so on.
Finally, I’m not a skeptic in the sense that I don’t think education is pointless. What’s pointless is trying to equalize outcomes without accounting for cognitive ability.
Thanks for chiming in, Educational Realist. You have many fans you don’t know about!
One issue that hasn’t come up yet is the importance of the early grades, and some sort of “launching” into a minimally adequate level of literacy by 2d or 3d grade. It seems to me that you could link this to the “Matthew Effect” whereby the best readers get better at an accelerated rate as reading become close to effortless (for some, at least).
Greg Cochrane was arguing that education in general hasn’t gone downhill–it’s just that we don’t have good time series data to show how little people had learned 50 years ago or 75 years a go.
I’m not certain that Cochrane is right. Some urban reformers seem to argue that literacy education (especially for Black boys) has deteriorated. I don’t think he’s reliable on this issue, but Walter E. Williams stated in _Up from the projects_ that most of his peers in the Philadelpia could at least read at some approximation of grade level. This is simply his anecdotal memory, alas.
I live in the suburbs of Rochester NY–one of of the previous Rochester City School District superintendents (Bolgen Vargas) was given to emphasizing that if the worst-performing kids weren’t reading at close to grade level by 2d or 3d grade level, they never caught up and simply fell farther behind as their cohort aged.
Vargas also was seen on video knocking on parents’ doors to track down their absent 1st and 2d and 3d grade children.
“Charters are basically private schools on the public dime, which means they can cap and skim. Both will increase the costs of education. Charters can only increase the amount we spend on education, which is crazy.”
Huh?
I’m curious what you’re basing this assertion on. So I can see how they work there, for what State are you most familiar with Charter Schools in? What’s the mechanism used to “skim”?
The Charter Schools I’m familiar with are required to accept all who wish to attend and if they run out of space are required to randomly determine who to accept.
I’ve heard rumors of some school administrations discouraging people from applying and others who may be swifter than most to kick out discipline problems, but as far as I can tell, those are a tiny minority.
My own experience is that where there are lots of Charter schools, most don’t hit their enrollment cap (and thus are only biased by parental choice/involvement) and that where there are few Charter schools, it’s primarily the parent’s who aren’t happy with the District schools who fight to enroll (leading to a flatter bell curve of students, top and bottom).
If you believe in a strong version of the null hypothesis, charters don’t improve things because nothing improves things. If you believe in a weaker version, that nothing government-run schools do can improve things (because of “capture” by providers, or bureaucracy or …), then maybe charters can break out and do better things that “establishment” organizations can’t. However, so far there is little evidence of that.
Good point about null hypothesis consistency. What charters do that no one discusses is make increase state expenses.
Charters were picked up by the right in the firm belief that they’d kill unionization and thus make teaching salaries lower. They were picked up by libertarians because libertarians don’t really understand how education works in America, or because they do understand and just don’t think it will be more expensive. But I’d argue that means they don’t understand.
I think you are right that libertarians wanted to break the unions, but my impression is that lowering wages was a minor part of the reason why. And even that has a relatively benign explanation. Many libertarians thought that in the “failing public school system”, teachers were unproductive and unsatisfied. In a better system that made full use of their talents, where they could see they were “making a difference,” they would both feel better and be willing to take less in their paycheck.
Much more important was, of all things, the auto industry. In the late ‘60s and early ‘70s, American cars were poorly built and expensive. The American Big Three seemed to be stuck in a culture of mediocrity, with a strong union and restrictive work rules that kept things from improving. But then, from thousands of miles away, came Toyotas and Datsuns. They didn’t break down as often, got better mileage, and were considerably cheaper. Japanese car companies had weak unions and a very different culture, focused on product quality. Perhaps American schools could do the same thing if “consumers” had a bigger voice and if the unions were broken. Perhaps American schools could have a focus on “student achievement.”
To many libertarians (and to most of my 11th grade students), the way teachers were compensated made no sense if the focus was on quality. Teacher pay depended on only two things, neither directly related to what students had learned: seniority and the amount of post-high school courses taken. A system with weak unions would do things much differently.
Ironically, the people who believed this shared a fantasy with the educational establishment. Said the establishment, “ Teachers are highly-trained professionals who can accomplish amazing things–if you spend enough money and pay them the salary they deserve.” Said the voucher/charter people, “Teachers could accomplish amazing things if they were just freed from the chains of public school bureaucracy and given a chance to shine, and to be paid according to how well they do their job (incentives matter!).”
This is the “weak version” of the null hypothesis I mentioned above. History has not been kind to it. Of course, the sixties had a lot of crazy ideas, some of which turned out to be not so crazy.
One of the things that many libertarians (and many non-libertarians) don’t understand about education is that the “buyers” are rarely the “consumers.” Young people are the “consumers” but the “buyers” are mostly parents or government officials. The “buyers” are, at best, experiencing the product at one remove.
In fact, the whole idea of “education” is that the “consumers” are ignorant. Even more that: not only aren’t they educated; they don’t know what education is. Someone else, who is educated, has to decide that for them. So the “buyers” will pretty much defer to the experts.
One of the reasons that vouchers/charters haven’t changed the system much is that most people accept that education is “what schools do,” or more precisely, “what good schools do”–with good schools defined as those where students get good marks and go on to selective colleges.
Charters are basically private schools on the public dime, which means they can cap and skim. Both will increase the costs of education. Charters can only increase the amount we spend on education, which is crazy.
That’s roughly how the higher education system works. And indeed the amount spent is crazy. Do you think it would be better if most people had to attend a local public college (and when I say “local,” I mean more like county than state)?
Or is college different in that skimming is bad K-12 but good 13+?
Higher education isn’t a right. I think we should limit higher education to those who have demonstrated a mastery of 12th grade education. And the state higher education system should focus resources only on those who are able to benefit from them. Which, as you know, is not what we’re doing now.
It is not hard for me to imagine some state legislature saying, “Nowadays, young people need at least some post-high school education to be successful. Therefore, we are raising the high school leaving age to 20 or an Associates degree. As part of our commitment to our citizens, this will be at no charge.” (Maybe they could even throw in, “Higher education is a right, not a privilege.”)
Assuming this happened and you were stuck with it, would you want to see a system like today’s or some sort of local 13-14 schools that pretty much everyone in the area was expected to go to? I.e., no moral equivalent of charters.
Oops, that should be “the school leaving age.”
” if they run out of space are required to randomly determine who to accept.”
Public schools don’t get to run out of space. Which is kind of my point. And expulsion from public schools is a long process, while expulsion from charter schools is an easy step, as no one has a right to attend a charter. Hence charters “skim” by expelling all the kids who aren’t well-behaved and failing all the kids who aren’t really prepared, so they can go back to the public schools. And the very act of applying for charter schools is a selector.
Of course, the more charters an area has, the more scrutiny they receive (cf New Orleans), and then the more like public schools they become, and the less advantage they’ll have over publics–making the whole exercise moot. I wrote about that here: https://educationrealist.wordpress.com/2016/08/24/charters-the-center-wont-hold/
Charles–thanks! But on that point, I agree with Cochrane. We didn’t do a good job decades ago, generally. And all evidence shows that early preschool doesn’t help much, even massive interventions like Perry. And Vargas got fired, didn’t he? Although they say they love him in New Hampshire.
Dear Education Realist,
Thanks for your comment.
I was hardly able to follow Vargas career, and these guys all blur together for me (I also lived in the Midwest, far from New York State, for two decades.) I don’t have much direct experience with the city schools, except for having gone to their alternative magnet high school (School Without Walls) back in the 1980s when it was still full of troublesome creative kids, many of them from the suburbs.
Regarding Vargas performance…it’s a difficult job and I’m not qualified to evaluate the people who fill the office. My sense is that Vargas was initially perceived as locally developed talent who had some commitment to the district, rather than as an ambitious leaper who would use Rochester as a springboard to a larger district. His predecessor, Jean-Paul Brizard, was able to leave Rochester for Chicago, for example.
Vargas doesn’t seem to have played his cards well while superintendent. I mention some of the things he said (about early success in reading and the uneven attendance of primary school students) because I suspect they are relevant to our discussion.
The null hypothesis is wrong. That’s trivial to show. But, for those caught up in the excitement of such a novel idea, let me provide an effective educational intervention.
I understand and speak French. Not well, it’s true, but certainly far more than I would have done, had it not been for the educational intervention called “French class”, without which I would speak perhaps a dozen words of the language (which happen to have been adopted by English or are commonly used in movies, like “oui”). This educational intervention has been rolled out quite widely, and has produced at least modest Francophone capabilities in millions of students who would otherwise have had little or no ability in the language (or even exposure to it).
That French Class works, to at least an extent, is a fact. That it is a widely-implemented educational intervention is also a fact. Educational interventions can be effective.
We can debate whether the intervention was cost-effective. But I, and millions of others, are proof that it is unquestionably effective.
It’s clear that the “null hypothesis” is wrong, but something closer to an efficient-market hypothesis may be correct. That is, most schools approach the boundary of efficient effectiveness, and adapt to stay there (the required adaptations are quite small, for the most part). Most innovations are, in fact, ineffective or overly-dependent on the passion of the implementers, or both; and the few that are effective and scalable are quickly adopted by the control group, rendering ineffective any attempt to identify them by comparison to control groups. And, with education having several thousand years of innovations behind it, most scalable innovations are only marginal improvements anyway.
Educational interventions are effective. But, scalable, effective improvements are hard; and measuring them is even harder. Still, that’s not at all the same thing as your null hypothesis.
Dear T Boyle,
Your point is a good one at some extreme limit of analysis, but it lacks analytical rigor.
Basically, we need a sample and we need controls.
1. we need a sample, not just your anecdote. Ideally we can get some experimental design.
For a sample, we could start with everyone else in your French class. Perhaps some of them now have French far worse than yours because they were unmotivated and had less of a God-given talent for language. Perhaps they now claim they have “forgotten it all.” Probably they never reviewed it in their head after that year.
2. We also need a control–the people who speak French as well as you do (or better) without having ever sat in French class a day in their lives, and who are no closer to “Francophonie” than you are (so they didn’t grow up in a French speaking milieu, for example).
= – = – = – =
Language proficiency is funny–understanding the word and speaking it tends to come naturally by acquisition for infants and toddlers. The developmental windows are important.
Reading and even more so writing seem to require some instruction.
A significant anecdote relates that Charlemagne never learned to read and write, though he had some proficiency in four or five languages.
H L Mencken is one of those who observed that it would be rare for someone to be able to write English with standard spelling without spending some time in a classroom.
When Arnold says “educational intervention,” I think he means some change that attempts to improve things. He is certainly not saying that people come out of 5th grade or 8th grade or 12th grade having learned nothing from school. He is not calling the existence of school and the requirement to attend an “educational intervention.”
I agree–mostly. But if you give it 20 or 30 years of aging, what’s the impact of 3 or 4 additional years of schooling, for someone who is 50 years old? Is it greater than (say) the person’s IQ or their ability to pass the “marshmallow test”?
We need a good definition of “intervention.” I actually think of a lot of interventions as “did you take the course, or didn’t you?”
There is a facebook meme, much shared, that says “Another day has gone by and I haven’t used algebra.” For me that’s stupid, because algebra helped me think in terms of multiple regression. I suspect that many people don’t think that way. I probably needed to take some courses–though in fact most of my thinking has come from reading, self-motivated, verbal presentations of applications.
Getting back to the notion of “interventions…”
These days we probably have “restriction of range” in the sense that the reasons not to go to college are drying up–or have dried up.
One thing I like about the Thiele Fellow idea is the notion that it creams off smart ambitious people and tells them to skip college. Maybe its a good idea, maybe not–but it does give us more data points.
Sorry if I’m beating a dead horse here. This was a good discussion thread.
I believe the test population would be a representative sample of folks who took French in school and the control would be a representative sample of folks who took about the same amount of French by other methods, like a community education course, Rosetta Stone courses, books at the library or visiting France for a period of time.
Do you think that the distribution of French speaking ability would be significantly different among those two groups?
Maybe Charlemagne could read and write–but with very poor handwriting. I did an internet search for “Could Charlemagne read” and found a page that asserts he could do so, though with effort.
The initial Charlemagne anecdote is found, methinks, in _Why our children can’t read_ by Diane McGuinness.
If you go to the West Hunter blog, the old blog entry entitled “Biology and Human Capital”, there is the assertion that John von Neumann seemed to know as much about Byzantine History as a professor of the field (at Princeton). Probably not from sitting in a classroom learning it.
There are people who absorb information more easily. Some absorb it slowly but retain it very well–such individuals may especially benefit from setting in a classroom or working through self-instruction programs.
I don’t know about T Boyle but most of my (uneven) language success comes from endless review in my head and practice with any tolerant person. Jakob Burckhardt said “We can never know too many languages. And however much or little we may have known of them, we should never quite let them lapse.”
Roger’s comments throughout are just bang-on.
Re teacher compensation: It seems obvious that paying teachers on the basis of student achievement is a “first best” system. But it turns out to be exceedingly difficult.
First, it has to be some sort of “value added” system. Ms. Agnes, who gets her 11th graders from a 6th to a 7th grade reading level is accomplishing more than Ms. Baltz, whose 11th graders go from 10 to 10.5. Ms. Clark’s students also gain a grade level, going from 10 to 11. But Agnes’s results are almost certainly harder to achieve than Clark’s. To be impolite, Agnes’s students are probably a lot dumber than Clark’s.
Then there is the question of how to vary compensation. Since Agnes’ students had advanced 6 grade levels in 10 years, they might be expected to gain 0.6 of a grade level in grade 11. But they gained 67% more. Clark’s students only gained half of what might be expected given their previous performance. Should Agnes get a 67% bump up from the base salary. Should Clark only get half the base? That seems kind of extreme.
And of course, there is always random variation year to year. Should changes to base pay only kick in if the actual achievement is 10% more or less than “expected”? 20%? 30%? 5%?
This all assumes that “reading level” can be easily and accurately measured–and that such assessment is done at the end of every school year. Of course, there would have to be achievement tests in every subject. And some sort of formula for courses that are in some way unique. How do you figure an “expected” grade in 11th grade physics? Average the grades in 9th grade biology and 10th grade chemistry?
Compensating on the basis of seniority and post-high school courses, along with a several year “wash out” period may be something like a “second best.” (It’s interesting to remember that the Japanese companies that ate Detroit’s lunch put a big compensation premium on seniority.)
During the first several years, many new teachers decide that teaching is not for them. They may feel they’re not good at it, that it isn’t worth the effort, that it’s just too unpleasant. They may have been told by a department head or assistant principal that their year to year contract will not be renewed. These are generally the worst teachers.. Of the ones that remain, some will be more interesting, some will be more caring, some will be more challenging. But in the spirit of the Null Hypothesis, they may all be about the same when it comes to student achievement. Seniority pay increases mean they have something better to look forward to every year. And since moving to another school probably means taking a pay cut, seniority leads to stability.
It’s hard to justify giving extra money for taking extra ed. courses, since so many of them are so awful or vapid. Perhaps a general morale thing. In effect, the school system is sending a message, “Our business is teaching courses and giving course credit. It’s so important that we’ll pay you more for taking courses and getting course credit.”