All relating to interventions with young children. Here, here, and here.
12 thoughts on “Against the null hypothesis, more research”
Study 1: ” Random assignment to a parenting program reduced disciplinary referrals. ”
Study 2: “Overall, our results reveal the importance of public programs and neighborhoods on human capital formation at an early age, highlighting that human capital accumulation is fundamentally a social activity.”
Study 3: “we find that Head Start generated large increases in adult human capital and economic self-sufficiency, including a 0.65-year increase in schooling, a 2.7-percent increase in high-school completion, an 8.5-percent increase in college enrollment, and a 39-percent increase in college completion.”
Study 3, Bailey et al on Head Start certainly seems to offer a new perspective. 35 percent increase in college completion seems huge.
The general rap against the Head Start program had been that effects fade away:
“While the Head Start program offered new and unique ideas about a child and their education in 1965, the program that still exists today is not yet modified enough to provide any significant benefits to a child. As the results have shown, all through the past 50 years, the effectiveness of Head Start findings has been fairly consistent in regards to the fade away effect. Starting in 1969, it was revealed that Head Start does not have all the lasting benefits that the public believed would end the cycle of poverty. With all of this in mind, one is left to wonder: if there is rarely positive effects found from Head Start, why is the program still intact today?” https://commons.trincoll.edu/edreform/2016/05/head-start-is-it-effective/#:~:text=Since%20a%20start%20of%20Head%20Start%20in%201965%2C,which%20at%20that%20point%20was%20serving%20561%2C000%20children.
Oddly the working paper doesn’t address the negative literature, only citing previous positive papers.
One wonders about the implications for all the state funded pre-K programs that have grown rapidly in the past 20 years. Florida, Georgia and Oklahoma and others now offer publicly funded “preschool for all ” for all 4 year olds (not just select groups) and participation rates are high in some. Perhaps comparative outcomes evaluations would be interesting.
At any rate Scott Alexander at Slate Star Codex had a lengthy post reviewing the evidence (https://slatestarcodex.com/2018/11/13/preschool-much-more-than-you-wanted-to-know/ ) in which he states “If I had to bet on it, I would say 60% odds preschool helps in ways kind of like the ones these studies suggest, 40% odds it’s useless.”
I found his logic persuasive so I don’t think I will let some new working papers move the needle on those odds.
Additional material against the null hypothesis from
Jonathan Chait:
“The ability of urban charters all over the country to get nonselective groups of poor, Black students to learn at the same level as students in affluent, middle-class schools is one of the great domestic-policy achievements in American history.
The fact that charters can produce dramatic learning gains is no longer in serious question.”
Charters are hardly nonselective, but they don’t exactly advertise it as such. They usually seem to select by putting enough hurdles in front of prospective and current parents and students to weed out the insufficiently motivated. IIRC Education Realist had several posts on this topic.
Chait references literature on lottery admission programs. Not sure how a lottery can be construed as selective.
It’s not like they generated a bunch of random street addresses and sent “if you have a 1st grade child, it is coming to our charter” letters. Chait writes about lotteries for people who had already applied in case the incoming class was oversubscribed. Also, I wrote “and current parents and students”. The Success Academy, Moscowitz’s chain of NYC charters which Chait mentions by name in that article, didn’t let students who were wearing wrong color pants, missing books and so on, to come into the school on school days, forcing parents to double back home to dress and equip them properly or take them to work. IIRC it had sort of report cards for parents, and it didn’t hesitate to put pressure on non-conforming parents to get them to switch to a different school if they didn’t tow the line. As you might imagine, insufficiently motivated parents don’t put up with such treatment for long.
You remind me that it was exactly Pondiscio’s book that Education Realist reviewed (and found lacking). The funny thing is that what I wrote above was Pondiscio’s theory of how Success Academy worked, but EdR writes that, to paraphrase, SA uses these rules to cull underperforming students even if their parents make valiant efforts to jump through all the hoops, and waives them for bright students whose parents don’t. EdR concludes:
—
Turn it around and posit that Pondiscio is completely wrong on this point and the data all hangs together nicely. Success Academy isn’t cherrypicking parents. They’re cherrypicking kids, just like the critics say. Kids who have a good chance of scoring proficient get to stay, even if their mom shows up drunk or their dad beats up the kids. Kids who won’t make the cut will get kicked to the curb, no matter how worthy their parents, how eagerly they comply with uniform, homework, and communication directives.
…
I have no proof of any of this, other than the data, which is manifestly inconsistent with a parental selection strategy, and Pondiscio’s own anecdotes, which clearly show that many parents aren’t meeting the very objectives he says Success is selecting for.
“As you might imagine, insufficiently motivated parents don’t put up with such treatment for long.“
Yes, I can see that discipline would encourage families that don’t want to get with the program to depart, yet, if discipline is the intervention that matters, why not take the outcome gains ? The undisciplined will not be any worse off.
There are also competing claims:
“This paper reviews 22 studies of enrollment issues related to student achievement, special education status, English proficiency, and student discipline in US charter schools. Of the 22 studies, nine use student level data to rigorously test for evidence of strategic enrollment. While the charter sector as a whole tends to serve fewer special education students and English language learners, there is much within-sector variation. Over all, there is very little evidence of systematic “cream-skimming” or “push-out” in US charter schools.
I guess I should be trying to figuring out what we can learn from what happens after Biden eliminates charters.
@edgar please see EdR’s review of Pondiscio’s book that I quoted above. EdR points out that at least for Success Academy attrition rates are quite high, the harsh rules of dress and behavior aren’t applied consistently, and concludes that the rules are used to remove underperforming students. Insofar as this is true, charter schools work by selection (which is what one would deduce from the null hypothesis) rather than by “intervention”.
Pondiscio sat in on many Success Academy classes, and interviewed teachers and administrators and parents, but never looked at student grades. So he could not rigorously compare the hypotheses, “SA is successful because it selects for motivated parents” and “SA is successful because it selects for more successful students”.
Since it is obvious that SA works very hard to make parents “get with the Program” (he gives lots of examples), he assumes it is the first.
There do seem to be quite a few charter schools which do not “skim the cream”, whose market niche is taking students who do not do well in their regular school. These also seem to be the charters which do not show gains in student achievement. This would be evidence for the null hypothesis. Alas, since “cream skimming” is illegal, it is probably impossible to get good data to test this.
And apprenticeships would appear to reject the null hypothesis as well:
“apprentices also mean cost savings for state budgets—and taxpayers. The Department’s recent $183 million grant to create more than 85,000 apprenticeships translates to $2,100 per apprentice. That’s much less than the $7,642 that the average state spends per student in college. In North Carolina, the state spends $9,959 per student. Encouraging high school graduates to become apprentices instead of chasing a bachelor’s degree could make state spending more efficient, and leave students better off.
In 2016, the Department of Labor counted about 21,000 registered apprenticeship programs like Siemens’ that train 500,000 apprentices across the country. Moreover, the Department reported that the average wage for an apprentice-trained worker is $50,000 per year—nearly $4,500 more than the U.S. median individual income and $300,000 more than non-apprenticed workers earn over the course of their careers.”
Gov’t support of business apprentices would be both individually good for those not academically oriented, and much much better for society.
As the supply of college-educated “middle-manager ready” elite wannabes swamps the demand for such paper pushers, good apprentice wages will reduce the gap with median, and mediocre, manager wages.
Study 1: ” Random assignment to a parenting program reduced disciplinary referrals. ”
Study 2: “Overall, our results reveal the importance of public programs and neighborhoods on human capital formation at an early age, highlighting that human capital accumulation is fundamentally a social activity.”
Study 3: “we find that Head Start generated large increases in adult human capital and economic self-sufficiency, including a 0.65-year increase in schooling, a 2.7-percent increase in high-school completion, an 8.5-percent increase in college enrollment, and a 39-percent increase in college completion.”
Study 3, Bailey et al on Head Start certainly seems to offer a new perspective. 35 percent increase in college completion seems huge.
The general rap against the Head Start program had been that effects fade away:
“While the Head Start program offered new and unique ideas about a child and their education in 1965, the program that still exists today is not yet modified enough to provide any significant benefits to a child. As the results have shown, all through the past 50 years, the effectiveness of Head Start findings has been fairly consistent in regards to the fade away effect. Starting in 1969, it was revealed that Head Start does not have all the lasting benefits that the public believed would end the cycle of poverty. With all of this in mind, one is left to wonder: if there is rarely positive effects found from Head Start, why is the program still intact today?”
https://commons.trincoll.edu/edreform/2016/05/head-start-is-it-effective/#:~:text=Since%20a%20start%20of%20Head%20Start%20in%201965%2C,which%20at%20that%20point%20was%20serving%20561%2C000%20children.
Oddly the working paper doesn’t address the negative literature, only citing previous positive papers.
One wonders about the implications for all the state funded pre-K programs that have grown rapidly in the past 20 years. Florida, Georgia and Oklahoma and others now offer publicly funded “preschool for all ” for all 4 year olds (not just select groups) and participation rates are high in some. Perhaps comparative outcomes evaluations would be interesting.
At any rate Scott Alexander at Slate Star Codex had a lengthy post reviewing the evidence (https://slatestarcodex.com/2018/11/13/preschool-much-more-than-you-wanted-to-know/ ) in which he states “If I had to bet on it, I would say 60% odds preschool helps in ways kind of like the ones these studies suggest, 40% odds it’s useless.”
I found his logic persuasive so I don’t think I will let some new working papers move the needle on those odds.
Additional material against the null hypothesis from
Jonathan Chait:
“The ability of urban charters all over the country to get nonselective groups of poor, Black students to learn at the same level as students in affluent, middle-class schools is one of the great domestic-policy achievements in American history.
The fact that charters can produce dramatic learning gains is no longer in serious question.”
https://nymag.com/intelligencer/2021/01/unlearning-democrats-answer-on-charter-schools.html
Charters are hardly nonselective, but they don’t exactly advertise it as such. They usually seem to select by putting enough hurdles in front of prospective and current parents and students to weed out the insufficiently motivated. IIRC Education Realist had several posts on this topic.
Chait references literature on lottery admission programs. Not sure how a lottery can be construed as selective.
It’s not like they generated a bunch of random street addresses and sent “if you have a 1st grade child, it is coming to our charter” letters. Chait writes about lotteries for people who had already applied in case the incoming class was oversubscribed. Also, I wrote “and current parents and students”. The Success Academy, Moscowitz’s chain of NYC charters which Chait mentions by name in that article, didn’t let students who were wearing wrong color pants, missing books and so on, to come into the school on school days, forcing parents to double back home to dress and equip them properly or take them to work. IIRC it had sort of report cards for parents, and it didn’t hesitate to put pressure on non-conforming parents to get them to switch to a different school if they didn’t tow the line. As you might imagine, insufficiently motivated parents don’t put up with such treatment for long.
Robert Pondiscio’s How the Other Half Learns: Equality, Excellence, and the Battle Over School Choice, A Year Inside America’s Most Controversial Charter School is generally favorable to Success Academy, but is very clear that the school will quickly get rid of students whose parents will not “get with the program”. One of the points of the book is that, though the system has a lottery for which students get in, the system makes it very clear that it expects students to act in certain ways and that those who do not should leave.
You remind me that it was exactly Pondiscio’s book that Education Realist reviewed (and found lacking). The funny thing is that what I wrote above was Pondiscio’s theory of how Success Academy worked, but EdR writes that, to paraphrase, SA uses these rules to cull underperforming students even if their parents make valiant efforts to jump through all the hoops, and waives them for bright students whose parents don’t. EdR concludes:
—
Turn it around and posit that Pondiscio is completely wrong on this point and the data all hangs together nicely. Success Academy isn’t cherrypicking parents. They’re cherrypicking kids, just like the critics say. Kids who have a good chance of scoring proficient get to stay, even if their mom shows up drunk or their dad beats up the kids. Kids who won’t make the cut will get kicked to the curb, no matter how worthy their parents, how eagerly they comply with uniform, homework, and communication directives.
…
I have no proof of any of this, other than the data, which is manifestly inconsistent with a parental selection strategy, and Pondiscio’s own anecdotes, which clearly show that many parents aren’t meeting the very objectives he says Success is selecting for.
“As you might imagine, insufficiently motivated parents don’t put up with such treatment for long.“
Yes, I can see that discipline would encourage families that don’t want to get with the program to depart, yet, if discipline is the intervention that matters, why not take the outcome gains ? The undisciplined will not be any worse off.
There are also competing claims:
“This paper reviews 22 studies of enrollment issues related to student achievement, special education status, English proficiency, and student discipline in US charter schools. Of the 22 studies, nine use student level data to rigorously test for evidence of strategic enrollment. While the charter sector as a whole tends to serve fewer special education students and English language learners, there is much within-sector variation. Over all, there is very little evidence of systematic “cream-skimming” or “push-out” in US charter schools.
https://www.tandfonline.com/doi/abs/10.1080/15582159.2017.1395614
This seems consistent with the relative low rates of reportable discipline in both charters and non-charters.
https://www.chalkbeat.org/2018/4/20/21104867/do-charter-schools-suspend-students-more-it-depends-on-how-you-look-at-the-data
I guess I should be trying to figuring out what we can learn from what happens after Biden eliminates charters.
@edgar please see EdR’s review of Pondiscio’s book that I quoted above. EdR points out that at least for Success Academy attrition rates are quite high, the harsh rules of dress and behavior aren’t applied consistently, and concludes that the rules are used to remove underperforming students. Insofar as this is true, charter schools work by selection (which is what one would deduce from the null hypothesis) rather than by “intervention”.
Pondiscio sat in on many Success Academy classes, and interviewed teachers and administrators and parents, but never looked at student grades. So he could not rigorously compare the hypotheses, “SA is successful because it selects for motivated parents” and “SA is successful because it selects for more successful students”.
Since it is obvious that SA works very hard to make parents “get with the Program” (he gives lots of examples), he assumes it is the first.
There do seem to be quite a few charter schools which do not “skim the cream”, whose market niche is taking students who do not do well in their regular school. These also seem to be the charters which do not show gains in student achievement. This would be evidence for the null hypothesis. Alas, since “cream skimming” is illegal, it is probably impossible to get good data to test this.
And apprenticeships would appear to reject the null hypothesis as well:
“apprentices also mean cost savings for state budgets—and taxpayers. The Department’s recent $183 million grant to create more than 85,000 apprenticeships translates to $2,100 per apprentice. That’s much less than the $7,642 that the average state spends per student in college. In North Carolina, the state spends $9,959 per student. Encouraging high school graduates to become apprentices instead of chasing a bachelor’s degree could make state spending more efficient, and leave students better off.
In 2016, the Department of Labor counted about 21,000 registered apprenticeship programs like Siemens’ that train 500,000 apprentices across the country. Moreover, the Department reported that the average wage for an apprentice-trained worker is $50,000 per year—nearly $4,500 more than the U.S. median individual income and $300,000 more than non-apprenticed workers earn over the course of their careers.”
https://www.jamesgmartin.center/2019/08/trump-moves-forward-on-apprenticeships-but-more-needs-to-be-done/
Gov’t support of business apprentices would be both individually good for those not academically oriented, and much much better for society.
As the supply of college-educated “middle-manager ready” elite wannabes swamps the demand for such paper pushers, good apprentice wages will reduce the gap with median, and mediocre, manager wages.