Overall, CMOs are delivering +.03 SD effects over three years in both reading and math. These gains are driven by the fact that students benefit from CMOs the longer they stay in them
In this context, CMOs are charter school management organizations, not collateralized mortgage obligations.
I still think that the overall picture tends to support the Null Hypothesis, although I believe that charter schools are capable of saving a lot of money while producing the same (null) effect as government schools. For a related meta-analysis, see Tyler Cowen’s post on a survey article on school vouchers.
.03 SD is just tiny. Example: for male height in the US, it’s just 2mm, less than a tenth of an inch, which is a 0.1% increase in the mean. Indistinguishable from “makes no difference in scores whatsoever.” Which is the null hypothesis.
Bit if it’s also cheaper, and produces more parent and student satisfaction, then it’s still an obvious big win, at least in those terms. Still leaves the obvious problem of having unfairly different de facto policy regimes and the risk of ‘creaming’ / flight, sending many marginal government schools into adverse selection downward spiral.
My takeaway would be they are successful at being no worse relative to that statistical power. As Arnold suggests, no worse and cheaper IS better. But also, no worse can also likely be improved upon.
Questions for you about your view of the “null hypothesis” in education.
I work in the health field, and myself believe the facts support something similar to the null hypothesis in health care. But there are two key ways in which I limit that hypothesis: (1) it applies only on average (i.e. there are health interventions that have both positive and negative impacts on health, and across a large population they roughly balance out), and (2) there is a small sick-to-very-sick subset of the population for which the net effect of health care interventions is positive.
The most difficult part of this worldview for me to reconcile has been to convince myself that the subset of negative-impact health interventions can have a large enough magnitude of an effect to counteract the health interventions that have a strong evidence base of a positive effect.
I’d appreciate if you could elaborate — or get similarly specific regarding any limitations — on your views of the education “null hypothesis”:
1. Do you believe that it is true only on average, or that virtually any/all educational interventions have no effect?
2. Contrary to #1, do you think that some educational interventions have a negative impact, and thus offset positive effects elsewhere? If so, what is the nature of those negative-effect interventions?
3. How do you reconcile your null hypothesis with the fact that children do learn, even if not enough or not the right things?
4. How do you reconcile your null hypothesis with substantially different educational outcomes among different populations (e.g. between nations, between socioeconomic classes, between two different public schools in the same city that serve a demographically similar population of students, etc.)?
This is taking the null hypothesis too far.
Arnold, are you of the belief that educational outcomes would be the same if we got rid of schools altogether? That almost seems to be what you’re implying here.
If schools do provide a benefit, are you of the belief that it is merely the presence of exams, and that students will individually do what they need to succeed on the tests independent of what the schools teach?
To put it in other terms: surely students who took statistics with you know statistics better than students who didn’t take statistics at all, right? The null hypothesis taken to the extreme would suggest that having taken your- or any- statistics course, should provide no benefit in understanding statistics at all.
Or are you contending that: given that students will need to take some exam – say AP statistics- they will ultimately get the same score regardless of how schools dish out the teaching.
If you search for Kling’s brief definition of the Null Hypothesis, you will see that it will automatically answer all those questions in an obvious way.
Unfortumately Arnold is not scalable or robust…in the context. There is also what I have dubbed the zeroth hypothesis. It would state that all that statistics you learned was only valuable within education.
Also, it is the null hypothesis, not the null theory, though I suspect there is one of those as well.
“To put it in other terms: surely students who took statistics with you know statistics better than students who didn’t take statistics at all, right?”
Immediately after the class, yes. But six months or a year later? Probably not if they don’t keep working with statistics. Robert Frank did some research with students who’d taken intro Econ a semester later. They performed no better than students who’s never taken an Econ course. And here’s a related take from Bryan Caplan:
http://econlog.econlib.org/archives/2012/02/the_career_cons.html
BD and Jeff,
I am fairly sure that when Arnold talks about the Null Hypothesis in education, he means the Null Hypothesis in education IN AMERICA. It is important to realize that there are very few really bad schools in America. No schools without roofs or heat. No schools where teachers don’t show up a few days a week or just read from a text. Few schools without texts or needed supplies.
You have a case here similar to the assertion that GRE scores don’t matter because success in grad school doesn’t correlate very well with GRE scores. But lots of people with low GRE scores haven’t been able to get into grad school, and probably wouldn’t have done as well as the people who got in. The people in grad school are the small upper part of the distribution. In that range, there is a Null Hypothesis in GRE scores.
(Below the level of US schools, there is a tremendous difference in what schools do and what young people get out of them. Lant Pritchett’s The Rebirth of Education: Schooling Ain’t Learning is an easy to read look at schooling and learning around the world.)
Homogenization of the educational experience is why the null hypothesis is true. Charters meet the same number of days, more or less the same number of hours, with teachers paid more or less the same as traditional schools. Why should we expect a difference?
Serious deviation from the traditional educational approach is the only thing that might lead to dramatically different student outcomes. Though, the null hypothesis at that point might convert over to a proven fact if the results still were consistent.
Null hypothesis in EDUCATION:
Although the word is used for other intentions, EDUCATION, in this context intends **collective** SYSTEMS for transfers of information, the means of its acquisition, its analysis and evaluations, and, its expansion along with the relationships of its various bits and pieces that constitute knowledge.
LEARNING is an ** individual** process for exercising the very same functions of those systems, whether or not it actually involves, or is involved in, any of those systems.
Most systems have evolved to the point of “course work” in which individual learning, regardless of individual capacities or differences, are required to absorb specific “packets” in specific segments within specific time or period allocations.
“Students” collectively, are to adapt to the prescribed course structures regardless of their individual capacities for absorption of the presentations and transfers in the prescribed time periods (and environments); resulting in a grading system for the individuals based on their relative absorption capacities or limitations, measured by the prescribed size of the “packets” and the periods specified for absorption.
All that may seem to be nothing more than rehashing the issue that some tend to learn some materials faster than others, some “retain” more of what is made available in the transfers. That has led in some instances to “assigned tracks within the standardized systems for the specifications of the “packets” and the allocations of time. Much of this has contributed to the null hypothesis among systems.
The Kahn Academy has opened an exploration into LEARNING processes by individuals (including within public schools) that returns to adapting the methods of transfers to the capacities and limitations (including motivations) of individuals, including the formats, especially sizes, for “packets” and the manner in which they are presented for transfer and remain available for absorption over periods adapted to the individuals. The results have been learning, rather than education, especially for many adults.
If the results from the explorations of the Kahn Academy continue to show their increasingly favorable results, and the areas for those formats of learning are not expanded too rapidly, we may be on the brink of a tremendous discovery that will end the discussions about the “null hypothesis in education.”
Soon you will have enough examples for a book that no one will want to read!
Since interventions are chosen that experts deem most promising, each example really ought to move beliefs. I suppose an optimist could reply that we are learning even more about where we ought to search so that new tests are actually more promising than previous tests, but this involves a large dose of incredulity.
I would like to know how far you see the converse extending. Just how little education and spending is required to produce the same learning? Probably we cannot entrust education entirely to cats and dogs, but…
Increased parental satisfaction, with similar test results, at a lower cost — this is a big win.
We should be trying more vouchers all over the US, but especially in communities where so many students are doing poorly.