From a couple of years ago, by Jon Baron of the Arnold Foundation (no relation).
Business: Of 13,000 RCTs conducted by Google and Microsoft to evaluate new products or strategies in recent years, 80 to 90 percent have reportedly found no significant effects.[iv]
Medicine: Reviews in different fields of medicine have found that 50 to 80 percent of positive results in initial clinical studies are overturned in subsequent, more definitive RCTs.[v] Thus, even in cases where initial studies—such as comparison-group designs or small RCTs—show promise, the findings usually do not hold up in more rigorous testing.
Education: Of the 90 educational interventions evaluated in RCTs commissioned by the Institute of Education Sciences and reporting findings between 2002 and 2013, close to 90 percent were found to produce weak or no positive effects.[vi]
Employment/training: In Department of Labor-commissioned RCTs that reported findings between 1992 and 2013, about 75 percent of tested interventions were found to have found weak or no positive effects.[vii]
Pointer from Michael Goldstein.
Maybe human beings are not fungible.
No they are not a very good observations.
This is the issue with the Lucas criteria. We do not scale, N, the count of agents, does not move at the speed of transactions and we never catch up with the myriad of demands imposed on us by government.
This is equivalent to the GDP factory problem, both restatements of the Lucas criteria. It is also responsible for our regular cycle of recessions nd a driver of inequality when the super wealthy have far more liquidity and can adapt.
Calculus tells us that the slope of any curve around a maximum approaches zero. One way to interpret the null hypothesis is to suggest that we’re already near a (local?) maximum in effectiveness and we’re testing small tweaks. We could test much larger changes if we wanted (e.g. punishing excessive student truancy with public execution, using opiates in recipes to secure brand loyalty), but those changes would be against the grain of our culture and disruptive almost by definition.
I think this is what Education Realist is saying. American education is about as good as it *reasonably* can be. Not “good” but “as good”.
Of course, that’s for a particular definition of “good”: essentially preparation for college. For at least half the population, that’s probably a crappy definition of good.
I agree with this….please apply to US foreign interventions.
An interesting natural experiment in education is the fact that instruction effectiveness over the past few months has seen a significant decrease, ”
https://www.msn.com/en-us/news/us/the-results-are-in-for-remote-learning-it-didnt-work/ar-BB155PAl
“Preliminary research suggests students nationwide will return to school in the fall with roughly 70% of learning gains in reading relative to a typical school year, and less than 50% in math …”
I think the Null Hypothesis guess would be that a year or so from now, most students will have scores that will be indistinguishable from the counterfactual, non-pandemic scenario.
It’s worth noting too that parents had this foisted on them against their will in the middle of a plague. It’s not like they wanted this or had a plan to make remote learning work. Mostly it just seems like they had have the kids watch TV while they worked from home.
Maybe the control group would be sending kids to public school in the middle of an epidemic. What would parents feelings on that be.
K12 is still a govt run apparatus that enjoys monopoly power over the 85% who can’t opt out of attending their local neighborhood school. Its no wonder than research done within that system shows no impact.
As for the research on charters and vouchers, yes, few studies show impact on central planners’ favorite measures (test scores)……But I don’t see parents leaving the schools they chose with their vouchers, tax credits, or charter lottery winning and running back to their old govt-run neighborhood schools.
The lesson here is that central planners don’t even know which measures matter to which parents. Thus, null hypothesis is a bit bogus…because the minute you pick a measure, you are acting like a [foolish] central planner.
Give people the means to choose whatever schooling they want for their kids….and butt out. Trial and error on the part of schools and parents will lead to more satisfied populace.
Of 13,000 RCTs conducted by Google and Microsoft, 10 to 20 percent have reportedly found significant effects. Reviews in different fields of medicine have found that 20 to 50 percent of positive results in initial clinical studies are affirmed in subsequent, more definitive RCTs.Of the 90 educational interventions evaluated in RCTs commissioned by the Institute of Education Sciences, close to 10 percent were found to produce strong positive effects. In Department of Labor-commissioned RCTs, about 25 percent of tested interventions were found to have found strong positive effects.
Does not sound good for the null hypothesis.